Recordings missing in Webinterface->Recordings, but found in "Removed"

[Langhuse]

Hi, I am back ?

 

Earlier I have reported missing files in the recordinglist, the root cause was never identified, but you updated your products and stopped using NTFS file properties. (link to post)

 

I did make a supervision script, checking the recordingdb (sqlite dbs) against reality on the file system.

This script kicks in from time to time

Most times it will get fixed  by saying "Task->Update Recording Database" I still does not understand why this is neccesary

 

Today I had a variant where Task->Update Recording Database did not work.

 

What I found was:

I was missing 4 recording in the webinterface->Recordings. They were not showing up when using the search box, but then I browsed the removed category (which is not searchable) and found all of them ?

 

 

The funny thing is that the Filesize > 0, where all other files in the removed section have a size of 0.

 

Just wanted to let you know

 

/poul

 

 

My environment:

- Mediaserver 2.1.6.1
- DVBViewer 6.1.6.1

WIN10

Recording stored on a Synology NAS.

Edited April 26, 2020 by Langhuse
typos

[Griga]

7 hours ago, Langhuse said:

The funny thing is that the Filesize > 0, where all other files in the removed section have a size of 0.

 

That's easy to explain and to reproduce. It means that the files in question were not visible for the last recording database update, but at least their file size is visible when  the details are displayed. The file size is not stored in the database, but retrieved on demand.

 

However, that leaves the question open why the files were (or still are) not visible for the recording database update. Some time ago there was a case where files could not be detected due to a corrupted "last modified" time stamp, which caused an error in the Media Server.  But recent versions catch this error and create a svcdebug.log entry.

 

Without this log (or even better a support.zip) it's impossible to tell if something similar happens in your case.

 

[Langhuse]

support.zip

Quote

It means that the files in question were not visible for the last recording database update, but at least their file size is visible when  the details are displayed.

 

Your comment "but at least their file size is visible when  the details are displayed." made me wonder if you have two different methods of accessing files - one sees them, the other do not

 

Note that it did not help to do a "Task->Update Recoring Database" !!!

 

 

Also attatched is a sqlite dump showing that that there are 5 files in recordings (the filename all starts with "Historien om Danmark"

INSERT INTO recordings VALUES(1095,'\\NAS3\video\Recorded\Historien om Danmark  Enev

 

Only 2 of these shows up under recordings, the 3 others are under "Removed"

 

I still do not understand

• why is it sometimes neccesary to do a "Task->Update Recoring Database" to recover files. Is the file change detection event driven except when doing a full scan?
• why can the recording DB contain the entries, but they are in the removed section. Is there something funny with how entries are categorised?

 

I have to say something is fishy, and can understand that it is not easy to figure out.

 

/poul - still believing that this is a good product ?

 

Note: You support.zip should include the DB and scan the recording folder to see what is there  (maybe you already do that). This would make analysis easier for you. But then again I might be the only one seen this, but I am pretty sure that this is a general (maybe small) problemt. Using a NAS for storage is not that uncommon.

db.1.dump.lst.zip

[Griga]

On 4/26/2020 at 11:38 PM, Langhuse said:

if you have two different methods of accessing files - one sees them, the other do not

 

Windows provides several methods for accessing files. That's what an operating system is supposed to do.

 

Windows API function for enumerating files in a directory:

https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-findfirstfileexa

https://docs.microsoft.com/de-de/windows/win32/api/fileapi/nf-fileapi-findnextfilea

 

Windows API function for getting the file size and other file attributes:

https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfileattributesexa

 

The svcdebug.log shows no errors on recording data base updates.

 

In the file systeminfo.txt I see two potential sources for problems in your system:

 

Quote

Installed Antivirus Products
--------------------------------------------------
Avira Antivirus
Avast Antivirus
Windows Defender

 

Which one is active? Is more than one active? Please note that security software may block file access. Try to disable it temporarily.

 

Quote

Public Firewall Profile is active

 

Your firewall is configured for a PC connected to a public network (public hot spot, internet café etc.), thus working very restrictively. It's definitely inappropriate for a PC used for multimedia purpose in a home network. This is not necessarily related to your problem, but may cause other issues. Besides correcting it I'd recommend to execute the files DVBViewer_Firewall_Rules.bat and DMS_Firewall_Rules.bat in the DVBViewer program directory  with admin rights for updating the DVBViewer/DMS related firewall settings.

 

On 4/26/2020 at 11:38 PM, Langhuse said:

• why is it sometimes neccesary to do a "Task->Update Recoring Database" to recover files. Is the file change detection event driven except when doing a full scan?
• why can the recording DB contain the entries, but they are in the removed section. Is there something funny with how entries are categorised?

 

The recording database update works like this: First it sets all database entries to "file not present" (removed). Then it scans your recording directories by using the FindFirstFile/FindNextFile API and sets the entries for all files that are enumerated by Windows to "present". The left over entries that are not found remain in the "not present" state.  This means, also the "removed" files are in the database. The difference is just a flag. The entries for "removed" files stay in the database until you delete them manually in the web interface.

 

Usually recording database updates (which can only be triggered manually) are not necessary because the database is automatically updated if new recordings are finished or old recordings are edited or deleted in the web interface. Updates are only required if something is changed outside the DMS.

 

[Langhuse]

Thanks for the comprehensive reply.

 

I will check out the antivirus/firewall. These programs sometimes (read often) result in difficult to understand scenarios - just like the scenarios I am seeing.

[Langhuse]

I did try disabling firewalls and all Antivirus and the “Rescan” did not fix anything.

I digged deeper by quering the DB and the FileSystem

Findings:

The SvcDatabase.db3 databases contains the “recordings” table.

• ·         The key is “idRecord integer primary key”

• ·         Filename is not unique “Filename text COLLATE SYSTEMNOCASE”. The filename includes the full path (not a good key, but normally unique on most filesystems).

Here is my first surprise because some identical filenames appear in two different records! Not forbidden, but could lead to trouble. What record will be used when finding a file while rescanning?

All fields are identical except

• ·         idRecord, this is to be expected as it is the primary key
• ·         enabled, ???

• ·         found, I guess that means it is on the file system

• ·         iLastModified, ??

3 out of 4 filenames with two records looks like this

• ·         Record 1: Enabled = true, found=true, iLastModifed=Number

• ·         Record 2: Enabled = false, found=false, iLastModifed=-1

• ·         These are all shown once in the web interface “recordings” (and not in the “removed” section)

1 out of 4 filenames with two records look like this

• ·         Record 1: Enabled = true, found=true, iLastModifed=Number

• ·         Record 2: same as Record 1, except iLastModifed is another number

• ·         This recording is shown twice in the web interface “recordings” (and not in the “removed” section)

You said the following in your last response

Quote

The recording database update works like this: First it sets all database entries to "file not present" (removed). Then it scans your recording directories by using the FindFirstFile/FindNextFile API and sets the entries for all files that are enumerated by Windows to "present". The left over entries that are not found remain in the "not present"

Q: how well does that work with two records with the same filename?

 

Some thoughts

• ·         Are two records with exactly the same name really intended?

• ·         Is it random whether a file is shown in “Recordings” or in “Removed”? Does it depend on whether the record with found=true or found=false is seen first?

• ·         What happens if a recording is restarted – could happen is the PC is sent to sleep while recording and the restarted (I have done that more than once)? The filename stays the same but is another record created?
• ·         What about my 3 original recordings in the “Removed” list with a file size > 0. The files do exist, the size is correct when shown, but the “Recordings” table in the DB says “found=false”. Who wins? the file on the FileSystem or the inconsistent DB found=false?
  You said in your last answer that you could see the size but not the file. But how can you pair the size with the correct fileName when shown in the Web if Recordings->Removed?

 

Maybe there are issues in the DB handling, maybe the problem is not the file system …

Don’t spend time on this if I am the only one with a small problem ?

Thanks for your time.

[Griga]

14 hours ago, Langhuse said:

enabled, ??? found, I guess that means it is on the file system

 

When a database update starts, all entries are set to found = 0. Then all found entries are set to found = 1. Finally, after all directories have been scanned, the DMS sets enabled = found for all entries. The decisive flag for the not removed / removed categorisation is "enabled". It is done in this way because the update process may get interrupted somehow, and this must not leave the database in an inconsistent state where many recordings are flagged as removed, though they are present.

 

14 hours ago, Langhuse said:

   iLastModified, ??

 

It's the "last modified" or "last write time" timestamp of the recorded file coded as integer (iLastModifiedTxt is the one of the associated EPG info file, which is an external representation of the database entry).The timestamps enable the DMS to recognize if a file has been changed in the meantime (e.g. cut or edited).  -1 means an invalid value, e.g. if the attempt to retrieve the "last modified" time via the WIndows GetFileAttributesEx API fails for some reason. When performing a database update the DMS performs the following steps for each file enumerated by Windows:

1. The DMS reads the "last modified" time for the path\filename reported by Windows from the database, if possible. If there is no entry with a matching path\filename, the DMS tries to create a new entry (see step 4).
2. If the path\filename is found in the database, but there is no matching EPG info file, though creating it is enabled in the options, the DMS recreates it from the database entry.
3. If the path\filename is found in the database, the DMS compares the "last modified" times from the database with the actual times reported by Windows. If both (for the recording and the associated EPG info file) are equal, it sets the database entry to found = 1. Otherwise it continues with step 4.
4. The DMS creates a new database entry or refreshes an existing one by reading the associated EPG Info file, if present. If this succeeds, the database entry is set to found = 1. Otherwise it remains in the found = 0 state. Media files without a matching database entry (path\filename and "last modified" time are equal) and without an associated EPG info file (same filename, but with .txt as extension) are not regarded as recordings by the DMS.

14 hours ago, Langhuse said:

Here is my first surprise because some identical filenames appear in two different records!

 

Maybe the explanations above help to find out how this can happen, though it shouldn't. I'll think about it...

 

[Langhuse]

Again, thanks for your time answering.

 

 

My SvcDatabase.db3 was pretty old, had around 1000 removed recordings, plus 3 wrongly in the Removed Recording section.

No stone should be left unturned!

So I thought - let us start out clean ?

The SvcDatabase.db3 is locked by the Media Server, so what I did was

• Stop Media Server
• Remove the db, but kept a backup ???
• Restart Media Server (creates a new DB automatically)
• Web Interface -> Task -> Update Recording Database, result 106 recordings

Surprise now I lost additional 26 recordings. Only 106 in the clean DB compared to 132 in the several year-old DB.

Then I did the following

• Web Interface -> Task -> Update Recording Database, same result 106
• Stopped media server and deleted SvcDatabase.db3
• Restarted media server (creates a new DB automatically)
• Web Interface -> Task -> Update Recording Database, same result 106. The 106 found is apparently not a random number
• Stopped media server
• Restored the old SvcDatabase.db3

   

• Started media server
• Web Interface -> Recordings -> All, still 132 recordings ?
• Web Interface -> Task -> Update Recording Database, still 132 recordings

I am deeply mystified.

The DVB media Server is unchanged and no changes to the filesystem and files. The single change was that the SvcDatabase.db3 was removed and a new fresh DB was created automatically by the Media Server.

 

 

Quote

When a database update starts, all entries are set to found = 0. Then all found entries are set to found = 1. Finally, after all directories have been scanned, the DMS sets enabled = found for all entries.

 

(my bold)

Q: How can the numbers of files found differ between an old filled DB and a new freshly created DB with everything else being equal?

 

 

Sorry for the mess   / poul

 

Additional info:

• I use UNC \\path for the NFS mounted nas
• The Media Server run as my user login (as system does not have access to the nas)
• Everybody else can se the recordings: windows explorer, DVBViewer, other video players, Visual Studio apps using System.IO

Edited May 6, 2020 by Langhuse

[Griga]

19 hours ago, Langhuse said:

Surprise now I lost additional 26 recordings. Only 106 in the clean DB compared to 132 in the several year-old DB.

 

What about the EPG info files that I've mentioned in my previous post? They play a decisive role in the update process.

 

On 5/5/2020 at 12:51 PM, Griga said:

Media files without a matching database entry (path\filename and "last modified" time are equal) and without an associated EPG info file (same filename, but with .txt as extension) are not regarded as recordings by the DMS.

 

Is there a matching .txt file for the "lost" recordings?

 

[Langhuse]

Yes there are a .TXT for all the recordings. We discused this in 2018 so I fully understand the importantance of these (link below)

 

I did check the following and found

• ·         It is not related to the NAS. I did a 4-hour copy to a local disk. The result was exactly the same

• ·         It is not the .txt files. I did copy a .txt file from a detected recording and used it to replace the txt file of a missing recording. This did not make the missing recording visible

 

I did write a small program (.NET) to travers the directory, it always finds all files, so does Windows explorer, DVBVierer for playback and other programs 

namespace DVBViewerFileList
{
    class Program
    {
        static void Main(string[] args)
        {
            DirectoryInfo dirInfo = new DirectoryInfo(@"\\NAS3\video\Recorded");

            Listfiles(dirInfo, "*.txt");
            Listfiles(dirInfo, "*.ts");
        }

        private static void Listfiles(DirectoryInfo dirInfo, string fileExtensions)
        {
            var fileList = dirInfo.EnumerateFiles(fileExtensions).OrderBy(x => x.Name).ToList();

            foreach (var file in fileList)
            {
                Console.WriteLine($"{file.FullName}");
            }

            Console.WriteLine(Environment.NewLine
					+ $"Number of {fileExtensions} files: {fileList.Count}"
					+ Environment.NewLine);
        }
    }
}

This always works finding all .ts and .txt files. I am sure that Media Server finds all relevant files.

 

The only explanation I see is that: Files/recordings are rejected due to TWO different Media Server conditions.

The model I see is that Media Server must works like this

1. Identify all videos with at corresponding txt file by traversing all paths defined in “Configure Server -> Recorder”

2. Make some sanity check – but what?
   1. Condition 1: Accept unconditional. Files existence is clearly not enough!

   2. Condition 2: If rejected by condition 1 then accept anyway if the filename is already in the DB (this could explain why my OLD DB is accepting more recordings than a new clean DB)

3. Update DB if accepted
   1. If record exist, refresh

   2. Create a new record

 

The big question is: what are the conditions for Media Server to accept recordings found on the filesystem?

 

Howto make  Media Server accept recordings otherwise rejected.

I have now automated the workaround I explained in 2014 (link below).

1. Rename the video file and the txt file by adding an “x” before the file extension “….x.ts”, “…x.txt”

2. Web Interface -> Task -> Refresh Recording DB

Voila ?

 

Other notes:

·         I assume that you got in control of the usage of the extended NTFS attributes in 2018 https://www.DVBViewer.tv/forum/topic/61265-recordings-missing-in-webinterface-recordings/?tab=comments#comment-472880

·         I assume that you traverse the filesystem first maybe followed by checked leftovers in the DB

·         I have had missing recordings since 2014 and created a workaround at that time, still works ?
https://www.DVBViewer.tv/forum/topic/55334-workaround-fixing-incomplete-recordinglist-in-webinterface/?tab=comments#comment-414931

[Griga]

On 5/6/2020 at 7:49 PM, Langhuse said:

so what I did was

• Stop Media Server
• Remove the db, but kept a backup ???
• Restart Media Server (creates a new DB automatically)
• Web Interface -> Task -> Update Recording Database, result 106 recordings

Surprise now I lost additional 26 recordings.

 

Please attach one of the .txt files belonging to the additional 26 lost recordings.

 

[Langhuse]

Here is one of the TXT files.

 

I did also check the extended attributes (as we discussed in 2018), the filename in there does match the actual filename, also after a rename.

 

I am still very suspecious about the "original filename" being blocked somehow. The receipe above where renaming always make the recording detected/visible does not work when trying to rename back to the orginal filename.

 

 

Letters from Iwo Jima_2020-03-27_22-05-00_DR2.txt

[Griga]

1 hour ago, Langhuse said:

Here is one of the TXT files.

 

It's ok, as far as I can see. Sometimes people edit these files with a text editor and save them in a different format, e.g. UTF-16 (Unicode with 2 bytes per character), so the DMS can't read them anymore. But your file is UTF-8, as it should be. I've checked it with Notepad++ and a hex viewer. It doesn't contain any invalid bytes.

 

1 hour ago, Langhuse said:

I am still very suspecious about the "original filename" being blocked somehow. The receipe above where renaming always make the recording detected/visible does not work when trying to rename back to the orginal filename.

 

Does this also apply if you start with a recreated (empty) database?

 

A possible assumption would be that a recording is no more detected if the filename is already in the database, but something is wrong with this entry, so it can't be set to "found" anymore. However, this is not possible if you update an empty database. There is no exclusion mechanism for recording filenames in the DMS, and there is no similar issue reported by other users.