Recordings missing in Webinterface->Recordings

[Langhuse]

First, thanks for some good products which I have used since 2011.

 

I have just upgraded to
- Mediaserver 2.0.4.0
- DVBViewer 6.0.4.0

...
and
...
an old problem came back. My list of recordings in the Web inteface now only includes 114 of 145 recordings.

 

What has happened so far

• moved recording folder to a new NAS (using a UNC-path for access)
• Upgraded DVBViewer and Media Server
• Moved a few recordings from the PC to the NAS (the wrong location was caused by Media Server installation which did modify the windows service login info)

 

Then I noticed that the list of recordings in the Web-Interface was mising some recordings.

I have reported this problem in 2014 see http://www.DVBViewer.tv/forum/topic/55334-workaround-fixing-incomplete-recordinglist-in-webinterface/?tab=comments#comment-414931

What is NOT working

1. Update Database (and also with variation of Clean Up Database and Clear Recording History)
2. Stop server, delete SvcDatabase.db3 and restart server

Trying 2. did in fact loose another 6 recordings

 

What is working

Rename the .ts, .log and .txt file followed by an "Update Database". I just add an "xxx" in front of the file extension.

 

This works exactly as reported in 2014 and the only difference seen from the Media Server is the changed filenames. My conclusion is the same as in 2014 that the filename is cached somewhere preventing correct detection.

 

A workaround

I can easily send a script through and rename all the files - but would rather learn what is wrong, so I will wait a few days before I do this.

 

Question/Way Forward

I could not find any debuging of the "Update Database" task. It would be nice to see why some recordings are rejected.

What should I do next?

 

Will a suppot ZIP include debug info from "Update Database"?

 

Edited April 3, 2018 by Langhuse

[majstang]

Are your NAS recordings the ones missing? Are you still on Windows 7? If you recently updated to Windows 10: 

http://www.DVBViewer.tv/forum/topic/59720-record-from-pc-to-a-nas-qnap-251/?do=findComment&comment=461690

If you already made that change before it can get resetted when installing a new DMS version.   

 

Edited April 3, 2018 by majstang

[Langhuse]

Thanks, for your input

 

I have been on WIN10 for several years.

 

Re: your link. - I have solved this issue several times.

Media Server Installation does not check the login currently used before using the System login for the service.

Maybe Media Server Installation should check the existing login and reuse it  when updating/replacing the service

 

The media server is better, now it used a WIN fallback(s) if it cannot open the UNC path, the old recording service did nothing and lost the recording you had to find the problem in the log.

 

The Problem - Recordings not accepted due to problem with the filename

 

The problem is still that the Webinterface Task "Update Database" will not accept a recording (.ts + .log + .txt) unless the 3x filename is changed to a new name.

 

Something is blocking - what?

 

A workaround:

1. rename the 3 recording files (.ts + .log + .txt) by adding a "x" before the file extension
2. "Update Database"

have to say that the workaround looks silly - but it works

 

 

Debugging

 

I am willing to help, but need a version of "Update Database" with debugging output first.

 

 

[Griga]

4 hours ago, Langhuse said:

Maybe Media Server Installation should check the existing login and reuse it  when updating/replacing the service

 

On 18.7.2017 at 7:03 PM, hackbart said:

Changes Media Server 2.0.3

• Change: General: The setup does not register the Media Server anymore on update installations (except if the installed version is the Recording Service), thus preserving configurations where the service logs on with user name and password for accessing network shares. For forcing a new registration the Media Server must be uninstalled first.

 

[Langhuse]

The change of system login did (as you guessed) happen when upgrading from the recording service to the Media Server.

I understand from your comment that this will not happen when upgrading the Media Server in the future - good will reduce user confusion a lot :-)

 

 

The Problem - Recordings not accepted due to problem with the filename

 

I made one additional test and did copy a recording that was not detected from the NAS to my PC (on a path included in recordings path list)

 

1. "Update Database"

Result: Not detected

 

1. Use workaround and modify the filename:
2. "Update Database"

Result: Success

 

Looks like the problem is not related to the NAS

 

Also, the database has the filename as a unique index. What happens when a file is COPIED to another path in the recording path list?

Edited April 5, 2018 by Langhuse

[Griga]

20 minutes ago, Langhuse said:

Also, the database has the filename as a unique index. What happens when a file is COPIED to another path in the recording path list?

 

No problem. Filename includes the path. Open the .db3 file with the SQLite browser and have a look at the content. It may also help to reveal the cause for the remaining problem.

 

On 3.4.2018 at 5:53 PM, Langhuse said:

What is NOT working

1. Stop server, delete SvcDatabase.db3 and restart server

 

Did you try that after upgrading to the DMS and correcting the service log-on setting?

 

[Langhuse]

Yes restarted many times after upgrading to Media Server, just did it again

 

1. stopped server
2. deleted all files in /Database
3. reboot PC
4. open WEB interface.
   No recordings in list
5. Task -> Update Database
   Recordinglist updated, but still missing the same 30 recordings (out of 140)

 

I am still mystified, change the filename and it works - why/what is blocking/cached?

 

[Griga]

There is no other caching of recording filenames / recording data in the DMS. Possible reasons for not inluding recordings on a database update are

• The path is no recording directory or one of its sub-directories.
• No EPG info file (.txt) and no NTFS file properties are associated with the file. The .log file doesn't matter.
• The file can't be found for some reason, e.g. because a NAS is offline or requires a log-on.

Is there something that only the 30 missing recordings have in common?

[Langhuse]

OK, good

 

Re bullet 1 and 3

• The path is no recording directory or one of its sub-directories.
• The file can't be found for some reason, e.g. because a NAS is offline or requires a log-on.

Yes the dir is in the recording path.

Also, why is it then detected when renamed.

Also still not detected when copied back to the PC (in a recording directory)

 

Bullet 2 -  Interesting

• No EPG info file (.txt) and no NTFS file properties are associated with the file. The .log file doesn't matter.

Again, the .txt is there and it works when the .ts and .txt are renamed.

...

but

...

what about the NTFS file properties ?

can I have lost these on the NAS or when copying to the NAS?

 

Maybe lost on the NAS but comes back when renamed on the NAS.

Also, maybe not created automatically when copied back to the PC until the files are renamed (could explain why the rename is also required on the PC)

 

- the properties looks normal when opened on the UNC path

- permission for user Modify/Read&Exe/Read/Write/Special but not Full Control

- tried to "touch .ts and .txt" on the nas side - still not detected

- tried to "touch .ts and .txt" on the PC side - still not detected

 

 

Can you advice how to check the NTFS file properties required for correct working?

 

 

Thanks for your help so far.

[Griga]

1 hour ago, Langhuse said:

what about the NTFS file properties ?

can I have lost these on the NAS or when copying to the NAS?

 

The EPG info file must be present or the NTFS file properties. If you have the EPG info file (and if the content is complete) the NTFS file properties are not needed. The NTFS file properties get lost if the file is copied to a non-NTFS drive, e.g. FAT32. What kind of drive format is used in the NAS?

 

1 hour ago, Langhuse said:

Also, why is it then detected when renamed.

 

I have no idea. If there is no other reasonable explanation, Putin is to be blamed. Ask Boris Johnson

 

[Langhuse]

You say

The EPG info file must be present or the NTFS file properties. If you have the EPG info file (and if the content is complete) the NTFS file properties are not needed.

 

"EPG info file" is this the .txt file?

If yes, then the txt files are complete and works (after the .ts and .txt is renamed).

A sample .txt file is attatched. Note that this particular recording was original detected after recording but disappeared in connection my setup of a new NAS and upgrade to Media Server (exactly when is unclear).

 

If not, what does "EPG info file" mean?

How can it be checked?

 

Sorry for the long thread ...

Ken Folletts Jordens søjler (1 4)_2018-03-29_18-15-01_DR2.txt

[majstang]

2 hours ago, Langhuse said:

"EPG info file" is this the .txt file?

Yes

 

NTFS file properties is the metadata (EPG details) attached to the actual recording.ts

 

The recording database rebuild read-in priority for DMS 2.0.4 is:

1. NTFS file properties (.ts)

2. EPG info file (.txt)

 

In next DMS version this order will be swapped, which may be favourable for you if having errors within the NTFS file properties.

Edited April 5, 2018 by majstang

[Langhuse]

Quote

 

1. NTFS file properties (.ts)

2. EPG info file (.txt)

 

 

All the NTFS and EPG looks correct for all my recordings.

Remember, they work when the recording is renamed!

 

I did write a program to extract the NTFS properties and made the following test

1. Copy a not detected recording to a new recording (adding "x" in front of the file extension)
2. Compare the files after the copy (no difference) (see below) - but this comparision does not include the NTFS properties
3. Dump the NTFS properties (see attached)
   The main difference is the "x" in the filename - as expected
4. Update Database
   Result: The recording with "x" is still detected the original recording is not  (see screendump - note the filename "<>x.ts")

 

using cygwin

The recording is copied to a new recording with an "x" before the file extension (last 3 files)

Files are compared "Files xxx are identical"

 

NTFS properties looks reasonable - see attatched.

 

 

Maybe MS$ is playing with us.

 

======================================================================================================

 

We have a problem …

 

I can fully understand that what I say does not make sense for you.

It does not make sense for me either – it should work – but does NOT.

 

I wonder why you haven’t used the oldest trick in the book – “please reinstall”.

Also, I wonder why you will change the order of using NTFS properties vs. EPG info file (.txt) in the next version – why do this if the code works flawlessly ...

 

My problem is likely a problem which shows up very rarely – and it is difficult to see even when it is there (it is always difficult to spot what is not there).

 

Let us stop here – and once again - thanks for a good product.

/poul

PS: I will I can make my workaround in a few days (just one command line but will not fix the core problem )

 

 

 

 

ts.properties.lst.zip

x.ts.properties.lst.zip

[majstang]

13 hours ago, Langhuse said:

Also, I wonder why you will change the order of using NTFS properties vs. EPG info file (.txt) in the next version – why do this if the code works flawlessly ...

 

Your reported problems are good proof enough the NTFS file properties handler do contain problems and is far from flawless. A memory leak has been detected and fixed in the new version, but the major drawback with the handler is it's affected by the DST (Daylight Savings Time). Although those changes of file time stamps doesn't affect rebuilding of the database any longer. The upside with reading the EPG info files (.txt) first is the content do not change over time. 

 

I don't think you will have to wait very long for the new DMS version and let's test this further then.    

Edited April 6, 2018 by majstang