TSD Operational Log - Page 14

Dear TSD User,

There will be a short maintenance window on Wednesday 22nd of August.

From 12:00 until 12:30 CET, there will be interruptions to the Connections with the network file system, and the shared folders will not be accessible for a short period. Please save your data before the maintenance window and follow our Operation Log for the update.

As soon as our engineers are done with the maintenance, we will update this Operational Log.

We are sorry for the inconvenience.

Kind Regards,

TSD@USIT

Dear TSD-User,

We will be performing a short scheduled maintenance of our Connection servers in TSD. Between 13:00 - 13:05 you will not be able to login to Windows machines. Users that are already logged in will not be affected by this downtime.

Dear TSD-users,

At the moment, Windows virtual machines in TSD are unable to mount \\tsd-evs. We are working in tandem with the storage team to resolve this issue as quickly as possible.

Our apologies for the inconvenience.

-- 
Best regards,
The TSD team

TSD is inaccessible for the moment. The engineering team is actively working to correct the issue.

Due to the planned power outage today, we had problems with the NFS-export, and therefore some jobs on Colossus were affected by this. The problem is now fixed, and we are sorry for the inconvenience.

TSD@USIT

Dear TSD users!

25.06.2018 @ 13:00 - 14:00 we will be performing a short scheduled maintenance of the central filesystem in TSD. Some of you might experience short hiccups in accessing your files during this time frame.

-- 
Best regards,
TSD team

Dear TSD users,

We regret to inform you, that due to sudden increase in concurrent Linux logins over the last two weeks, we've reached our limit for concurrent logins.

This means that some of you might not be able to log in to your Linux-VMs using the conventional method of https://login.tl.tsd.usit.no.

We have ordered more licenses and are eagerly waiting for them to arrive.

In the meantime:

You can log in to your Linux-VMs by first logging in to your Windows-VM using the following address: https://view.tsd.usit.no and then opening the application Putty to SSH to your vm. The hostname for your machine will have the following format, here using project p11 as an example: p11-tl01-l.tsd.usit.no

-- 
Best regards,
TSD

Some projects could not login to their windows machine through view. The incident took place during the time slot:

Start:          2018-05-13 13:00
End:            2018-05-14 10:00

We apologize for the inconvenience

TSD@USIT

Dear TSD users,

We are currently experiencing some troubles with the FileLock
We are working hard to find a solution.
Our sincere apologies for the inconvenience.

-- 
Best regards,
TSD-team

Dear TSD users,

We regret to inform you that we are currently experiencing some troubles with the nettskjema-integration for TSD.

Due to this, not all nettskjema answers will be copied over to your import folder at the moment.

They are still stored securely and encrypted on one end of the chain, and we are working as quickly as we can to figure out what has gone wrong.

Lately, the usage of Colossus has increased, and now usually every node is running jobs.  This triggered a problem in the queue system, which basically prevented projects not already having a running job to get their jobs started.

We have now developed a local fix to the queue system so that this no longer happens.  The patch was applied today, and seems to work well. This means that now all projects should be able to get jobs started.

The most user visible change is that if you use the command "pending" to view your project's pending jobs, the top jobs should now get an estimated start time (most of the time, at least).

We have recently detected problems with file import/export, and are currently working on solving this.

Dear TSD user,

The upgrade process is taking a bit longer than expected, and we are now working on the security checks. Due to this, we are extending the maintenance window to 16. March.

We are sorry for the inconvenience.

Dear TSD user,

As announced earlier, we will start doing maintenance on the virtual desktop infrastructure in TSD on Wednesday the 14th and Thursday the 15th of March.

This will not be a complete downtime for TSD. We will be working to upgrade the Windows servers and the VMware Horizon login infrastructure. This will be done in an incremental fashion over these two days. Please do not schedule any long running jobs on your Windows servers during this time. The consequence of the maintenance is that it will be impossible to log in to your project' s Windows server. This will probably only last a few hours per any given project. Linux VMs will not be affected. Colossus jobs will not be affected either.

At the same time we plan to perform an upgrade to a newer version of PostgreSQL. Projects running this database should expect some downtime on Wednesday.

These changes will help us provide better service to yo...

Dear TSD users,

Unfortunately, we had a network outage last night, and this is causing some issues with login and mounting of file systems in TSD at the moment.

We are hard at work resolving the issue as quickly as possible.
Our apologies for the inconvenience.

--
Best regards,
TSD

The file system on Colossus crashed at around 13:10 today. We are currently working on solving the problem as quickly as possible.

UPDATE:

The file system went down due to a fuse going. We have investigated and moved cables around to make sure that we do not get a repeat of this incident. Our apologies for the inconvenience.

Currently, jobs on Colossus have problem starting. They seem to hang in "CONFIGURING" (CF) state, but eventually start after some minutes. We are investigating the problem, and will come back with more information when we know more.

We are sorry for the inconvenience.

Update 14:28: The problem is fixed, and the jobs are starting as normal. Thank you for your patience.

Dragen is down due to security patching

Dear TSD user

Due to a serious security vulnerability in modern processors, which in practice affects all operating systems, and therefore almost all IT services, we need to perform maintenance on the entire TSD infrastructure. This means that all TSD services will be affected. Please do not start any critical work during this period.

We will try our best to complete the work tomorrow between 09:00 and 17:00 but due to the scope of work and the short planning horizon it is possible that parts of TSD will remain under maintenance until Monday 8th of January 17:00.

If you want more information please consult UiO security [1]. This has also been widely covered in Norwegian press [2,3]. Those who want more detail about the vulnerability can refer to more technical explanations [4,5].

Regards

Leon du Toit

[1]...

Dear TSD user

We are performing security related maintenance on TSD infrastructure on Thursday, 28 December 2017. From 09:00 to 17:00 it will not be possible to log in to TSD, or to initiate new data transfers through the file-lock. Active colossus jobs will not be affected.

We will post information about progress on the 28th on the operational log.

Apologies for the inconvenience.

TSD@USIT

We are experiencing issue with https://view.tsd.usit.no and users trying to login to Windows VMs through this method will not be able to get in.

Sorry for the inconvenience and we will fix this as soon as possible.

Unfortunately we are seeing some issues with some of our Linux VMs.

We are hard at work looking for the root cause of the issue.

If you are experiencing trouble with your virtual machines, please contact us at tsd-drift@usit.uio.no and we will do what we can to get things back in order as quickly as possible.

Our sincere apologies for any inconvenience.
TSD@USIT

Due to a security issue in the last version of Thinlinc, we must upgrade the Thinlinc infrastructure tomorrow 28/11-2017 between  13:00-15:00 CET.

During the operation, the login to the Linux virtual machines will not be possible. The windows users will instead only experience three short outages, during which the machines will be hanging for a short while. All the process running on both linux and windows VMs might die out. Our best advice is then to log out from the machine as soon as the maintenance window will start.

Please follow the operation on our operational log.

We apologise for the inconvenience.

TSD@USIT

Jobs that will not end before 28th November will be placed on a pending state and will start automatically after 28 Nov.

e.g.  as of now (20th Nov) if you ask for a time limit of less than a week the job should start. Otherwise the job state will show as PD....