Jump to content

RS Database Winter time change bug


majstang

Recommended Posts

There seems to be a nasty wintertime change bug that affects the RS database. I have 360 dummy.ts and dummy.txt files in my Database backup folder. When clearing recording history database i did notice i only got 181 entries upon refreshing DB. I then deleted ths svcdatabase thinking there was some corruption and restarted RS and refreshed DB and now there's lots of entries missing in the recordings database. Looking at the metadata (from the explorer) the mutual thing with the missing entries is durationtime has disappeared from the metadata and been replaced by 00:00:00. The old enties before the winter time change that actually shows up when refreshing recordings DB has altered duration time. One can notice some peculiar numbers has been added onto the duration time looking like this:

512409509:46:10

 

The actual stats from txt file for this example are:

[General]

Version=1.1

[Media]

Created=29.10.2011 09:04:06

Channel=Discovery Channel

[0]

Id=110290815

Date=29.10.2011

Time=08:15:00

Duration=00:50:00

Title=Mythbusters

Info=E48

 

Man o man...looks like i have to correct all 360 entries manually :(

Hopefully this one can be fixed, cuz my database will be much larger for the summertime change ;)

Link to comment

No there is for sure no timechange bug. The summer/winter time does not affect already present recordings.

 

For testing purposes I rebuild my recordings database and I can not confirm any problems.

Link to comment

Ok, it was just a guess the winter/summertime change could cause this issue. The actual corruption of the recordings db and recordedlist may very well have happened earlier, but since not having cleared recordings history db and refreshed recordings db in a while i didn't notice any problems until doing that after the winter time change.

 

As i have understood it when deleting the svcdatabase and rebuilding it, the durationtime will be retrieved from the recordings properties (fileinfo/metadata) and not the EPG information file (TXT). Checking the recordingsdatabase and recordings history database there's lots of negative values for the durationtime. All these entries with negative values has the strange durationtime value described in post 1 (512409509:46:10) when looking at the fileproperties in the explorer. However these entries does show up at least in the recordings db and recordedlist. The problematic part is the ones that have for unknown reasons got the durationtime zeroed, these ones does not show up at all and the consequence of that is shows that i have recorded before are being recorded again. 182 dummy.ts are now missing from the recordings db (with durationtime 00:00:00) and are now useless :(

 

Starting to wonder if the recordings properties for explorer application, which i chosed to install when installing RS, is the culprit here. Things gets even more complicated cuz i do use a script making the dummy.ts and then using propcopy to transfer the recordings properties from original fullsize recording to dummy.ts. Original recording gets deleted when watched. It have been working flawless many months, until now. The script backs up these dummy.ts and txt to three different locations over two harddrives. Checking all dummy.ts on every backup location i have the same kind of fileinfo corruption everywhere, so it's not limited to the RS subfolder which i use for main backup. That may point to some problem with propcopy, but this theory can't be right cuz dummy.ts created several months ago, which i know for a fact were fully functional for rebuilding the database before, has now been corrupted.

 

Lars, perhaps i can mail you the everything i have on this if you wanna have a look at it?

 

Regards

Majstang

Link to comment

Ok, i finally found some massive proof this is a wintertime change bug by using the delprops tool. FileInfo view was identical to the Info text File before wintertime change. After it the fileinfo view has changed and there's an hours difference. Notice how this affects the duration time...it goes completly haywire. Negative values in svcdatabase and strange numbers in explorer view.

post-56433-0-69150100-1320356197_thumb.jpgpost-56433-0-14803500-1320356226_thumb.jpg

 

Also made some screendumps on the zeroed out durationstimes...i.e those files/entries never show up when rebuilding the database.

post-56433-0-03994500-1320356243_thumb.jpgpost-56433-0-67410100-1320356258_thumb.jpg

 

Not one single corruption of any dummy.ts that was created after the wintertime change has occured. Only affected files were created before 30/10.

Edited by majstang
Link to comment

After more investigations i now have a slightly clearer picture what happened. Seems the corruption of the fileinfo has going on continuously since 2011-09-24. All dummy.ts created between May and 24 of September got the fileinfo "Date" changed 1 hour back at the 24th. After this date the fileinfo date has been changed automatically when the recording was created up to 30/10. After the 30th of October the change of fileinfodate stops. Now the information txt file and fileinfo date corresponds with each other.

The consequence for this one hour backward reset, one can see on the fileinfo durationtime.

All shows with the duration time of 1 hour subtracted with the resetted hour equals 01:00:00 - 01:00:00 = 00:00:00

In other words all shows 1 hour long got the durationtime zeroed out, more than half my database disappeared because of this error.

Other shows half an hour long did show up, but with a screwed up durationtime: 00:30:00 - 01:00:00 = 23:30:00

 

Thank god for the delprops application. I have now restored the fileinfo on all my dummy.ts and the database is back on track again. No fun maunally going over several hundreds of recordings.

 

The only thing missing now is the most important thing...why this happened...i have no idea!? Any ideas? Unfortunately my svcdebug.log doesn't stretch that far back anymore, so no clues there. I will definitely keep an watchful eye on this one in the future.

Edited by majstang
Link to comment

Did some lookups about Windows summer/winter time handling. My Windows location setting is UTC+1 (wintertime). When summertime comes Windows handles it via the automatic DST correction. The UTC+1 changes to UTC+2 (summertime). The only explanation for this issue above, i can come up with, is Windows stores all file time stamps in UTC+1 regardless of DST correction. Since im an hour ahead of UTC+1 during summertime the FileInfo.LastWriteTime property doesn't care much about the DST correction, it just assigns LastWriteTimeUTC with the value returned from DateTime.ToUniversalTime(). That should be why the fileinfo "Date" always is stated in UTC+1 format, despite it is actually UTC+2 time. There's probably nothing one can do about this handling from Windows programatically and one have to simply accept it.

 

However the very thing sinking the whole RecordingService database is how the Fileinfo.length coding re-calculates the duration time using the UTC+1 time value and subtracting it with the actual duration time value. This must be possible to stop somehow. So my question to Lars is can anything be done about this? Stopping the re-calculation of duration time would render it possible to actually use the prevent recording of re-runs function without hassle. Not being forced to manually fix the fileinfo on all recordings made during summertime.

 

Cheers

Majstang

Edited by majstang
Link to comment

Did some lookups about Windows summer/winter time handling. My Windows location setting is UTC+1 (wintertime). When summertime comes Windows handles it via the automatic DST correction. The UTC+1 changes to UTC+2 (summertime). The only explanation for this issue above, i can come up with, is Windows stores all file time stamps in UTC+1 regardless of DST correction.

 

Not exactly. Windows always stores all dates and times internally in UTC only, and then offsets your current timezone setting, e.g in your case +1 during winterime, and +2 when DST is active. Probably makes no difference for your case, but still thought I'd clear this up.

Link to comment

Not exactly. Windows always stores all dates and times internally in UTC only, and then offsets your current timezone setting, e.g in your case +1 during winterime, and +2 when DST is active. Probably makes no difference for your case, but still thought I'd clear this up.

 

Now that's cool. I just noticed that while all my recordings still work, my RS Database went totally nuts after the recent DST change:

 

1. All Recordings pre- DST end in Germany are off by one hour. E.G recordings started at 23:00 are shown as 22:00.

 

2. Recordings after DST end are displayed just completely nuts. I programmed two recordings (via DVBViewer epg itself, not through the web interface of the RS) this saturday, one at 18:30, and one at 23:15. Both recordings *were* executed at those times, but the web interface of the RS shows the first one starting at 19:00, the second one at 21:15. I have *no* idea how this is logically possible, but it's nonsenic.

Link to comment

Not exactly. Windows always stores all dates and times internally in UTC only, and then offsets your current timezone setting, e.g in your case +1 during winterime, and +2 when DST is active. Probably makes no difference for your case, but still thought I'd clear this up.

Thanks for clearing this up and reporting about your troubles as well! The problematic part is the fileinfo handling as i see it. Do you have much knowledge about this? Since the Recording Service database retrieves most of the important values about recordings from the fileinfo, it is very frustrating discovering the fileinfo handler do not take the slightest consideration to the DST correction and that should be why we see all these troubles we have described here. Thankfully the fileinfo handler does seem to correctly handle when DST is deactivated at least.

 

Now that's cool. I just noticed that while all my recordings still work, my RS Database went totally nuts after the recent DST change:

 

1. All Recordings pre- DST end in Germany are off by one hour. E.G recordings started at 23:00 are shown as 22:00.

 

2. Recordings after DST end are displayed just completely nuts. I programmed two recordings (via DVBViewer epg itself, not through the web interface of the RS) this saturday, one at 18:30, and one at 23:15. Both recordings *were* executed at those times, but the web interface of the RS shows the first one starting at 19:00, the second one at 21:15. I have *no* idea how this is logically possible, but it's nonsenic.

Yes, i did notice this as well making me realize something was seriously wrong. I also did notice the issues you are describing in 2, but after fixing the fileinfo on all recordings everything seem to work fine again. Maybe the corrupted values you have in the database snowballs into other issues.

Edited by majstang
Link to comment
  • 4 months later...

Just cleared the previous recording history and deleted the svcdatabase (have to do it that way with this RS version) in order to rebuild both RS database and recording history. I did it to test if I got the same trouble with the fileinfo/properties as after the wintertime change (when the database went haywire and most of it disappeared). Results was disappointing sadly enough. As expected all fileinfo on the .ts recordings recorded before 2012-03-25 (visible in the balloon tooltip and the file properties in the explorer) got the length value added by 1 hour. A 1 hour show are now shown as a 2 hour show. All recordings made after the 25th has correct values.

 

Now, contemplating the consequences if doing nothing since the lenght value plays no vital role (in theory) for the workings of the "check for previous recordings function". Simply leaving the wrong length untouched will render upon the next wintertime change, the value will go back to correct length (1 hour subtracted), except for those recorded after 2012-03-25. Most of these last ones will get thier length zeroed out and disappear from the database (if rebuilding it) like described earlier in this thread and the nightmare starts again completely sinking the "check for previous recordings function".

 

Doesn't any solution exist for this annoying issue?

Please do tell if there's any ideas how to fix it ;)

If there isn't why not remove the lenght value from the fileinfo/propeties all together until there is a solution at hand?

 

I think this is a major problem when it causes so much trouble for the "check for previous recordings function". When having hundreds and hundreds of recordings in the database it's a daunting task manually changing the wrong values in the fileinfo every time when going from summer- to wintertime, in order to get RS back on track.

 

regards

majstang

Link to comment

I found the problem. :blush: Consider it fixed. With the next release the start times and durations will be correct if read from the fileinfo.

 

BTW I really hate your avatar, it makes me want to swat at my monitor... :D

Link to comment

I found the problem. :blush: Consider it fixed. With the next release the start times and durations will be correct if read from the fileinfo.

Makes me very glad to hear...many big thanks to you Lars :thumbsup:

 

BTW I really hate your avatar, it makes me want to swat at my monitor... :D

Haha...well it is a bit like me...an annoying little bugger :rotfl:

Link to comment

Well I have a two monitor system. on the left screen is the delphi IDE, on the right screen your forum post. And I see this annoying little bugger at the edge of my vison when I'm looking at the IDE and try to work. I had to minimize the webbrowser because I couldn't really concentrate with this thing running around. ;)

 

 

A short explanation of the problem. The code is quite old and it seems I made some wrong assumptions when writing it. In fact I'm quite emberassed thinking about it now.

But since this problem is present in the fileinfo of recordings since like forever there is not much to fix the main problem, only to work around it.

And the workaround for the duration field is nothing to be proud of, believe me. :(

 

But for future recordings I have a simple solution which will integrate nicely and avoid the issue completly.

 

I've integrated the changes in the RS and the DVBViewer, which is also affected.

Link to comment

The code is quite old and it seems I made some wrong assumptions when writing it. In fact I'm quite emberassed thinking about it now.

But since this problem is present in the fileinfo of recordings since like forever there is not much to fix the main problem, only to work around it.

And the workaround for the duration field is nothing to be proud of, believe me. :(

No need to feel embarrassed, this is complicated stuff. It is impossible to foresee every little angle and how it will work with future code. If wanting an easy career you should have chosen driving a truck or something ;)

This workaround will be great, thanks :)

 

But for future recordings I have a simple solution which will integrate nicely and avoid the issue completly.

Only downside is, i suppose, this means i'll have to manually go over the database with delprops to get the correct fileinfo durations back :( Well, no probs since it will be the last time doing so.

 

Best regards

majstang :bye:

Link to comment
  • 1 month later...

A short explanation of the problem. The code is quite old and it seems I made some wrong assumptions when writing it. In fact I'm quite emberassed thinking about it now.

But since this problem is present in the fileinfo of recordings since like forever there is not much to fix the main problem, only to work around it.

And the workaround for the duration field is nothing to be proud of, believe me. :(

 

But for future recordings I have a simple solution which will integrate nicely and avoid the issue completly.

RS 1.9.5

Fix: Recording: (Re)reading the file info of recordings had problems with Summer/Winter time.

System Win7x86, RS 1.9.6, DVBViewer Recording Properties v.1.0.

Unfortunate this workaround didn't fix the problem all the way entirely. Instead of changing the fileinfo on existing database entries (dummy.ts backup) it now does the same re-calculation of the durationtime and creates duplicate entries when rebuilding the database. One with the actual and correct durationtime and one with 1 hour added to the durationtime. This happens to all entries recorded before the summertime change. Instead of the expected 700 entries I get 1100 entries. Perhaps it is possible to have a look at it again?

 

I have described the problem in this thread also:

http://www.DVBViewer.tv/forum/topic/48856-DVBViewer-recording-properties/page__view__findpost__p__361385

 

Regards

Majstang

Link to comment

Just to make it clearer picturing this database rebuild bug:

 

Correct entry:

Show duration - 00:30:00

Start and stop time - 17:30-18:00

post-56433-0-87955000-1336315753_thumb.jpg

 

Duplicate entry:

Show duration - 01:30:00 (one hour gets added)

Start and stop time - 18:30-20:00 (start time gets added by one hour)

So it seems there's three issues if counting this duplicate entry being created in the first place.

post-56433-0-76742500-1336316026_thumb.jpg

 

Db backup fileinfo:

post-56433-0-97592300-1336316118_thumb.jpg

Seemingly still correct fileinfo data and no changes or additions gets done amongst the db files, which is good.

 

Info file (txt) of the entry in question:

[General]

Version=1.1

[Media]

Created=30.12.1899

Channel=Kanal 5

[0]

Id=7595

Date=16.06.2011

Time=17:30:00

Duration=00:30:00

Title=Rules of engagement

Info=S01E05/7

Series=Rules of engagement

Description=Amerikansk komediserie från 2007. Jennifer och Adam planerar en romantisk helg. Jeff och Audrey funderar på att skaffa barn. Russell träffar på en äldre kvinna från sitt förflutna. Skådespelare: David Spade, Megyn Price, Patrick Warburton, Bianca Kajlich, Oliver Hudson, Gästskådespelare: Jessica Walter. Säsong 1. Del 5 av 7.

Charset=255

Content=0

MinimumAge=0

TimerID=

[stats]

Errors=0

Size=1022,36 MB (1072023752 bytes)

Avr. Datarate=0,533 MB/s

Device=7231 BDA DVBC Tuner (9)

Edited by majstang
Link to comment

Found more strange results after rebuilding the database.

Besides the duplicates I also get strange durationtimes in RS Db despite the fileinfo durations is ok on dummy.ts. Don't know where RS gets these durations from when the database gets rebuilt (obviously not taken from the fileinfo or info text file anyway). Here's some dumps:

 

Entry 1:

Durationtime - 23:30:00 (one hour subtracted from correct duration 00:30:00)

Starttime - 20:00 (correct starttime)

Stoptime - 19:30 (one hour gets subtracted from stoptime)

post-56433-0-66709300-1336602689_thumb.jpg

 

Fileinfo on db file is correct

post-56433-0-61296800-1336602733_thumb.jpg

 

Info text file

[General]

Version=1.1

[Media]

Created=25.03.2012 19:59:02

Channel=TV4

[0]

Id=203252000

Date=25.03.2012

Time=20:00:00

Duration=00:30:00

Title=Kontoret

Info=E07/8

Series=Kontoret

Description=Svensk komediserie från 2012. Ove har övertygat Linköpings universitet att han ska få komma dit och föreläsa. Dessvärre krockar föreläsningen med ett möte gällande besparingar på kontoret med hans chef. Men Ove tänker inte missa sin chans att få prata inför studenter och gör allt för att smita från mötet. Viking upptäcker att man går miste om många saker när man inte äter lunch med kollegorna. Del 7 av 8.

Charset=255

Content=0

MinimumAge=0

TimerID={9E8AEB48-097F-4BE1-8B15-71A09FE87717}

[stats]

Errors=0

Size=1,04 GB (1117937300 bytes)

Avr. Datarate=0,556 MB/s

Device=7231 BDA DVBC Tuner (3)

 

 

Entry 2:

Durationtime - 23:30:00 (one hour subtracted from correct duration 00:30:00)

Starttime - 19:00 (one hour gets subtracted from starttime)

Stoptime - 18:30 (two hours gets subtracted from stoptime)

post-56433-0-78196100-1336602983_thumb.jpg

 

Fileinfo on db file is correct

post-56433-0-33859800-1336603007_thumb.jpg

 

Info text file

[General]

Version=1.1

[Media]

Created=18.03.2012 19:59:02

Channel=TV4

[0]

Id=203182000

Date=18.03.2012

Time=20:00:00

Duration=00:30:00

Title=Kontoret

Info=E06/8

Series=Kontoret

Description=Svensk komediserie från 2012. Ove har lovat att ordna en avskedsfest för en medarbetare som gått i pension. Något han helt glömt bort och nu försöker han få Therese att ta på sig ansvaret för den. Viking upptäcker att alla på kontoret är medlemmar i en hemlig facebookgrupp förutom han och Ove. Erik har svårt att koncentrera sig på jobbet på grund av renoveringsarbeten. Del 6 av 8.

Charset=255

Content=0

MinimumAge=0

TimerID={2E076479-AACE-4994-9F6F-B0C677D5B9F2}

[stats]

Errors=0

Size=1,04 GB (1117478392 bytes)

Avr. Datarate=0,556 MB/s

Device=7231 BDA DVBC Tuner (3)

Edited by majstang
Link to comment
  • 2 weeks later...

The issues has been reduced after going over my system, which is a bit overcomplicated at the moment. Turns out the duplicate issue was my misstake entirely. RS did pick up additional db entries from one of my backup drives when rebuilding the database :blushing:

 

The automatic DST correction dependant duration re-calculation issue in the fileinfo has been fixed from RS 1.9.5. This fix takes care of no DST correction could eliminate the check for previous recordings function any longer, which is great and many thanks for that :original:

 

The only remaining issue, which is a minor one cuz it's merely cosmetic and can't break down any vital functionality, is the DST correction still messes with start and stop time calculation for the recordings. Start and stop time gets re-calculated according to the examples in the posts above and displays rather comical in the RS database. Wondering therefor if it is possible to exclude already recorded entries from the DST correction and re-calculation of start and stop time? Keep it exclusive for timers and internal/process tasks?

Edited by majstang
Link to comment
  • 5 months later...
A short explanation of the problem. The code is quite old and it seems I made some wrong assumptions when writing it. In fact I'm quite emberassed thinking about it now.

But since this problem is present in the fileinfo of recordings since like forever there is not much to fix the main problem, only to work around it.

And the workaround for the duration field is nothing to be proud of, believe me.

After the wintertime change the database is still complete after rebuilding it and the check for previous recording function works nice, despite the DST correction messes heavily with the duration field causing strange duration times in the fileinfo of recording entries and recording durations in RS database webinterface (which is calulated based on the fileinfo). I did post about yet an another duration field bug yesterday:

http://www.DVBViewer.tv/forum/topic/50771-rs-recordings-editor-file-info-bug/page__view__findpost__p__375471

 

Yes, I believe you, now when seeing the results the duration fileinfo implementation (workaround) is causing and it's a mess. However the important functions works as is now, but barely. It's in a dire need to be rewritten thouroughly and reimplemented properly.

Edited by majstang
Link to comment
  • 2 years later...

Hi guys! Wintertime change still causes issues although not as severe as before. This area may need an overhaul soon. The current issue is the prevent recordings of re-runs (check against) does not work right after the winter DST. Its an easy fix though. Clear and Rebuild Recordings History and delete all timers (done before DST) and redo the autotimer. Then its up and working again. It would be great if you could get this working more reliable without any manual actions from user side :original:

Link to comment
×
×
  • Create New...