Merge Duplicates causing Hoster to not respond

[stefbeme]

Yesterday I added new songs to my database. I rebuilt the database and I have now lost thousands of files. Upon rebuilding it is stating I have 30000+ duplicate files. When I click on the merge files button Hoster does nothing, it locks up and will not proceed. I have version 5.10 I have checked my settings according to your guidelines for Windows Performance Changes and they are correct. I use Windows 7 on the existing computer that is having this problem. Please advise.

[Roy Dennis]

The not responding may be due to the amount of duplicated files mentioned.

How did you add these new files to the database?
How many new files did you add ?

Was it from the Build database dialog using the Add folders button, Index Hard drive files dialog or the auto index new files feature.

Some things you can check, have a look in your Quarantine folder first to see if any of the files have been moved to there you can find that from the Tools Menu/Hoster Data Folder, Quarantine.
There are also a couple of text files in the data folder you can check should tell you the file path of where the files were "BuildSongErrors.txt"
Then look for "MergeSongDBLog.txt" file and open it. If you look at the date header that corresponds to the date you tried the merge and see if any songs are listed.

[stefbeme]

Originally yesterday, I added about 20 files to an existing folder, I rescanned the folder were the files were added. Then (of course during a show) I get a 911 from my husband that there were many songs missing. He went to tools>songs database>rebuild database>no resolution>I then rescanned each individual folder>no resolution>I then looked at my excluded folders>added a few folders that were questionable>no resolution at this point the amount of merge files has now grown, each time I did each of the above steps. When I attempted to merge, Hoster would then freeze (not responding). We rebooted, attempted to merge several times, each resulting in freezing Hoster.
I am assuming after reading your reply, that I am (and have always) been adding new files incorrectly so I may need the best direction on how to accomplish this.
I have searched the Quarantine folder, and no files have the song title, numbers only therefore am unable to determine if the files are included in this folder.
MergeSongDBLog.txt>the only file that exists is dated March 21, 2013 which does not correspond with my attempt on 8/9/13.
BuildSongErrors.txt - the missing files are located here and I have identified there are MANY files in this txt that are not appearing in my current listing in Hoster (this gives me hope)

[Roy Dennis]

First off I see nothing wrong with the way you added new songs providing they were new songs that had not been imported before, example on another computer.
If they were songs that already had a kmh header attached or a kma file imported using another computer this could cause conflict.

Are all your music files on an external drive ?
If they are first off I should exclude the C:\ drive, make sure there are no folders showing in the list of folders for the C: drive.

Have you at some time added songs using another computer ?
If so adding tracks this way can confuse Hosters Book Id or Disc ID numbering as one computer doesn't know what ids were used.

As there are no files listed in the "MergeSongDBLog.txt" file is probably because none have been merged.

You need to identify the folders the duplicated files are in. Open the "BuildSongErrors.txt" file and it should show something like: Book ID, Title, Artist, File Path, and and Type (Kma or KMH), note down the main file path excluding the title artist part, are most of them showing the same file path, in other words are the majority from the same folder ?
Hoping they are as this can make things easier.

Next note down a Book ID and type that book id into Hosters Book id field, does another song show for that same ID ? if so that is where the conflict is, add that song to playlist, right click on song in playlist and select edit from the dropdown.
This will show that song in the edit dialog, note the Disc Id, if it's a true disc Id copy it from the black disc id window at the bottom noting the track # then using Windows explorer paste that into the search box of the drive containing the song files and see if it finds that track, if it does note down the folder that the song is in.
If the disc Id is not a true disc Id like XYZ0xxx then you will have to just search on the title which can make it harder to identify if you have multiple choices but if you play the song from the playlist you should be able to recognise its brand thereby helping identify the track.
Now repeat the above with a couple more of the songs listed in the .txt file and see if the majority are in the same folder.
Hoping that they are exclude that folder for now from the rebuild list.

Now you say the files in your quarantine folder are all just numbers, I would assume in that case that they are kma types, if so open Hoster Tools Menu/Hoster Data Folder/Quarantine folder, do the files numbers correspond to any of the disc Id's in the .txt file?
If they do close Hoster but leave the quarantine folder open.

You need to copy these files back to their original folder, If they are kma types I would assume they are in one folder so this should be no problem.

Assuming things are as I stated and not masses of problem folders first rename the .txt file just in case we want to refer to it again for backup purposes, Hoster will ignore it if you rename it, start Hoster and rebuild the database making sure the problem folder or folders have been excluded, also make sure none of the "Song Types To Index" buttons are selected so that it will only index the previously indexed files. If there are still problem files it will re-create the .txt file.

See how you get on with that first then come back it may be you will get a few duplicated files but hopefully a lot less than previously.

If that goes alright I'll try and guide you through re-importing the problem folder.

[stefbeme]

Open the "BuildSongErrors.txt" file and it should show something like: Book ID, Title, Artist, File Path, and and Type (Kma or KMH), note down the main file path excluding the title artist part, are most of them showing the same file path, in other words are the majority from the same folder ?
Yes
Have you at some time added songs using another computer ? Yes, when I add songs to another computer I load them onto a flash drive, then add to a folder on that computer>rebuild database. We can look at this possibly later if this is causing confusion. I was not aware the two could speak to each other.
does another song show for that same ID ? No In fact much to my amazement, there are only about 120 songs showing in my Hoster>search
I have copied all the kma files from Quarantine to the C:/MTU/KMA files (existing file folder)
I am uncertain which is the exact problem folder so I chose to exclude two folders, rebuilding the main one with the KMA files.
Here are the result:
18340 Duplicate ID
8 files failed to index in the Database.
Click the ‘View Log File’ button to find them
on the hard drive.
You may need to re-import bad CD tracks,
or fix the bad
original mp3+g, wma+g, zip, audio, video files.
This contains file information:
Error file: C:\MTU\KMA Files\MMVE20-1 (2).kma
Error file: C:\MTU\KMA Files\MMVE20-1.kma
Error file: C:\MTU\KMA Files\PS1275-4 (2).kma
Error file: C:\MTU\KMA Files\PS1275-4.kma
Error file: C:\MTU\KMA Files\SC3458-9 (2).kma
Error file: C:\MTU\KMA Files\SC3458-9 (3).kma
Error file: C:\MTU\KMA Files\SC3458-9.kma
Error file: C:\MTU\KMA Files\SC9014-6.kma

Again, when I attempt to merge the duplicate file it has stalled Hoster (not responding)

And by the way I do want to mention thank you for your assistance on a weekend. It is very much appreciated!!!!!

[Beavis]

its not responding because its working on merging. it may take hours.

[stefbeme]

Patience is a virtue! Thank you, it has now merged files

Awaiting further instructions

[Roy Dennis]

Originally Posted by stefbeme

In fact much to my amazement, there are only about 120 songs showing in my Hoster>search
I have copied all the kma files from Quarantine to the C:/MTU/KMA files (existing file folder)
I am uncertain which is the exact problem folder so I chose to exclude two folders, rebuilding the main one with the KMA files.
Here are the result:
18340 Duplicate ID
8 files failed to index in the Database.
Click the ‘View Log File’ button to find them
on the hard drive.
You may need to re-import bad CD tracks,
or fix the bad
original mp3+g, wma+g, zip, audio, video files.
This contains file information:
Error file: C:\MTU\KMA Files\MMVE20-1 (2).kma
Error file: C:\MTU\KMA Files\MMVE20-1.kma
Error file: C:\MTU\KMA Files\PS1275-4 (2).kma
Error file: C:\MTU\KMA Files\PS1275-4.kma
Error file: C:\MTU\KMA Files\SC3458-9 (2).kma
Error file: C:\MTU\KMA Files\SC3458-9 (3).kma
Error file: C:\MTU\KMA Files\SC3458-9.kma
Error file: C:\MTU\KMA Files\SC9014-6.kma

Again, when I attempt to merge the duplicate file it has stalled Hoster (not responding)

As beavis mentioned it can take a time to merge.
Is the 18340 the amount it indexed or the amount of duplicates ?

If you look at those 8 files they are all copies of one another except the last one, the figure in brackets at the end denotes it's a copy of the one without the brackets, the one SC3458-9 has 2 extra copies (2) & (3) if a lot of your files are like that you must be using lots of hard drive space. I should look in your kma folder and see if you have lots of these with the extra figure in brackets.

Anyway now that those kma files have merged do you think you are seeing all those 18340 files in your search dialog now, only you stated above you were amazed to only find 120 ?

When you import your files on your other computer do you import them as kma or kmh types ?

Is your other computer using version 5.10 ? if not what version is that using ?

Are the files in the folders that you excluded kma or kmh types ?

Do you still need to import the files in those two folders you excluded ?

[stefbeme]

The 18340 Duplicate ID were the amount of duplicate files that have now been merged, I know can see them in my KMA file.

When I import files on my second computer, they are a zip file on a flash drive, I copy the files onto an external hard drive, then import using KMA option.
My second laptop is Windows 8 with version 5.08
Are the files in the folders that you excluded kma or kmh types ? These are Kmh files and yes I still need to import these songs please.

[Roy Dennis]

If you just Add the folders to the rebuild list I think you will just get another load of duplicate book id's again so I would import them again from scratch from the import song files dialog.
Before attempting this open the folder in Windows explorer and delete the "ImportedFiles.txt" files important. You can also delete any other .txt files in this folder and all the .kmh files so you are only left with the original zip file or if they are mp3+g the mp3 and the cdg file.
This can be made easier by right clicking in the folder and select sort by Type, if type is not showing select More then tick the box for type then that option should appear.
Now you need to open the Build database dialog, select the excluded folders button, select the folder in question and remove from list to allow it to be imported again.
Now use the index songs files button to open the Import Hard drive files dialog on the actual computer your having the problem with. Depending on the way your files are named it could be best to set the field separators to ignore the Disc Id and track # unless you specifically want that info imported, this will allow the full 99 tracks per book id and avoid any conflict if the files in this folder have an identical disk Id to those you already have imported.

The problem with importing from different computers arises when you import some files one week on one computer, then another time choose to import on the other computer, if you were to stick to the same computer every time the problem would be unlikely to occur.

When you import songs Hoster allocates a book Id and a disc id if one is not entered. So if you were starting from scratch the first book Id would be 101 which would be disc 1 track 01. Now after many imports you may have 50101 (Disc501 Tr 01) now if you get another computer and you import a new disc in it and import it will be given the 101 number, so if you then try to copy this already imported file to the other computer it sees it as a duplicated. You can set a higher number than the one allocated when importing where you see the next button but you can still get a problem with disc id numbers if they end up the same.

[stefbeme]

It has been a journey, but I have now indexed all remaining files. There were quite a few! It seems that all my songs have been restored!

I am now ready for my next step which I believe would be resolving the conflict with my second laptop regarding the ID numbers, correct?

What is the best process moving forward when I obtain new files? Add them to my main computer first, then what?

Also you are very correct. My KMA file has so many duplicates in it. The file itself holds 32.9 GB. What would be the best process to reduce the size of this file? Remove all the files that have a (2), (3), or "copy of" or would it be best for me to send you that file (if possible) for you to review?

[Roy Dennis]

Originally Posted by stefbeme

It has been a journey, but I have now indexed all remaining files. There were quite a few! It seems that all my songs have been restored!

I am now ready for my next step which I believe would be resolving the conflict with my second laptop regarding the ID numbers, correct?

To do that you would need to make the data base an exact replica of the one you just rebuilt.
This would mean removing all the existing files from the other computer.
I would then rebuild the database to create an empty database, then make sure none of the Song Types to Index buttons are highlighted so that it will only index files that were already indexed and not look for any others. Copy all the folders from the one you just rebuilt to the other computer, add those new folders to the rebuild list, you should see it add each folder to the database.
I'm still a bit confused as to whether you store the actual song files on each computers C: drive or on an external hard drive ?
In post #9 you stated "When I import files on my second computer, they are a zip file on a flash drive, I copy the files onto an external hard drive, then import using KMA option".
When they or on the hard drive they are zip files so do you just keep them on the external for backup purposes and just have the kma files on your C: drive ?
Or do you store the imported kma file on the external as well ?
If you do the later that means you are doubling up on hard drive space.
Also wondering as you seem to be getting all these duplicated files if you have this external hard drive attached containing the zip files when you rebuild the database with the zip types and the kmh buttons selected which would then start trying to re-index the zip files that you already imported as kmh types. It might even try importing them again as kmh types.

Originally Posted by stefbeme

What is the best process moving forward when I obtain new files? Add them to my main computer first, then what?

I would say providing you always import the files using the same computer then copy the newly imported file which in your case sounds like the kma files to your flash drive then paste them into the identicle folder on the other computer and rescan that folder there should be no problem.
I personally prefer to import them as kmh types because then you don't double up on hard drive space you just need to store the small kmh file with the zip file

Originally Posted by stefbeme

Also you are very correct. My KMA file has so many duplicates in it. The file itself holds 32.9 GB. What would be the best process to reduce the size of this file? Remove all the files that have a (2), (3), or "copy of" or would it be best for me to send you that file (if possible) for you to review?

It would probably have been best to have deleted al the (2), (3), or "copy of" files before we started trying to recover your files. I don't think there is much I could do by reviewing the file.
If you are going to delete them best to do it before you try matching the two databases up or forget it now that it's merged them all.

If you decide to delete the (2), (3), or "copy of" files in Windows explorer then I would rebuild the database again after else Hoster will still think those files exist.

[stefbeme]

After doing my first show since we have done all the above work, I am sad to say that I am still missing multiple songs. I really feel like the best process for me would lbe to start from scratch for both computers. What would be the best process for this?
Computer #1 has Hoster 5.10 Windows 7
Computer #2 has Hoster 5.08 Windows 8

I have one file with KMA's, the remaining songs I have a zip file for these.

When I review my KMA files, there are some files marked with an extension of "pk". What does this mean and do I need to keep these files?

[Roy Dennis]

Originally Posted by stefbeme

After doing my first show since we have done all the above work, I am sad to say that I am still missing multiple songs. I really feel like the best process for me would lbe to start from scratch for both computers. What would be the best process for this?
Computer #1 has Hoster 5.10 Windows 7
Computer #2 has Hoster 5.08 Windows 8

I have one file with KMA's, the remaining songs I have a zip file for these.

When I review my KMA files, there are some files marked with an extension of "pk". What does this mean and do I need to keep these files?

The pk extention I have never heard of or seen with a kma file.
I just googled the pk extension and it comes up with quote "The PK file type is primarily associated with 'Audition' by Adobe Systems Incorporated. Adobe Systems Incorporated acquired the technology assets of Syntrillium Software in May 2003. On August 18th, 2003 Adobe released a rebranded version of Cool Edit Pro 2.1 as Adobe Audition software. Audition/Cool Edit creates .PK files which are known as Peak Files. Peak files store data associated with the graphical representation of audio data (i.e., Waveform diagram). Audition/Cool Edit has the option to disable the creation of these files, however this may slow loading, saving and re-drawing of graphical data. These files can safely be deleted and will not effect the original audio data in any way."

So have you at any time used any of those programs ?.
It does say it's safe to delete them if that's what they are.

When you re-imported them last time, did you set the field order to ignore disc id and track as I suggested ?

Are all your kma's, files that you imported directly from disc ?

Only in post #9 you stated: "When I import files on my second computer, they are a zip file on a flash drive, I copy the files onto an external hard drive, then import using KMA option."
What worries me here is if you imported them as kma files and you re scan that folder a second time with the kmh and zip buttons selected you are effectively importing the same song again as kmh file and it would register as being imported into the songs.mdb even though you might remove that drive from the computer afterwards, if it did this it would see the file as a duplicated file and remove the previous imported one as the duplicate thereby making it so that song no longer appeared.
Just speculation on my part but trying to understand whats happening in your case.


Truly starting from scratch would mean re importing them from CD so I don't suppose you want to go to that extreme.

What we did before was almost like starting from scratch apart from your existing kma files, you did say you had a load of duplicate with the (2)(3) numbers, you would need to manually delete these in Windows Explorer and any other files that are not kma files.

As you can't alter any info inside a kma file these need to be Added to the list of folders first as they have all their disc id and book id data inside them.


To start from scratch I would do the following assuming your still using version 5.10
Open the Tools Menu/Hoster Data Folder then close Hoster but leave data folder open, find the "HosterInfoIndexed.ini" file and delete it.
Now start Hoster and go to the rebuild database dialog, your list of folders to index should show a blank list, rebuild the database now and it will build a blank empty data base. If the list is not blank select whatever is showing in it and see if any files show in the right hand pane, you don't want any songs to show in it at this point.
Now assuming you have edited all the duplicated (2) &(3) copies from the kma folder and there are no other type of files except kma's in the folder, Add that folder to the list of folders making sure first none of the "Song types to index" buttons are selected.
Hoping this time it does not show too many duplcates.

If you do get any duplicated notices after adding those merge songs before continuing.
If you get loads like before report back before continuing as you need to get this folder right first.

For the KMH types:
As before open the kmh folders in Windows explorer and delete the "ImportedFiles.txt" file, important. You can also delete any other .txt files in these folders and all the .kmh files so you are only left with the original zip file or if they are mp3+g the mp3 and the cdg file.

Now use the index songs files button to open the Import Hard drive files dialog. I would set the field separators to ignore the Disc Id and track #, this should stop the problem if any of the files have identical id's to those in you kma folder or any other folders. Proceed to index as normal.

I can't think of any other tips if others can join in.

[stefbeme]

To start from scratch I would do the following assuming your still using version 5.10
Open the Tools Menu/Hoster Data Folder then close Hoster but leave data folder open, find the "HosterInfoIndexed.ini" file and delete it.
Now start Hoster and go to the rebuild database dialog, your list of folders to index should show a blank list, rebuild the database now and it will build a blank empty data base. If the list is not blank select whatever is showing in it and see if any files show in the right hand pane, you don't want any songs to show in it at this point.
Now assuming you have edited all the duplicated (2) &(3) copies from the kma folder and there are no other type of files except kma's in the folder, Add that folder to the list of folders making sure first none of the "Song types to index" buttons are selected.
Hoping this time it does not show too many duplcates.
I have completed this process. There are 2409 duplicate files. But I also noticed there are over 14,000 files in that folder, it only imported 1994. This does not make sense. I may want to start with scratch with the 500+ CDG's. It is 2 weeks before my next show.
What are your thoughts?

[Roy Dennis]

Originally Posted by stefbeme

I have completed this process. There are 2409 duplicate files. But I also noticed there are over 14,000 files in that folder, it only imported 1994. This does not make sense. I may want to start with scratch with the 500+ CDG's. It is 2 weeks before my next show.
What are your thoughts?

Are all the files in that folder kma type files ?
Only my instructions should only import the kma types regardless of how many files in that folder.
If that folder also contains other types let us know what types.
If they are all kma types I'm afraid I'm stuck for an answer.

At a rough calculation 500 discs would only be around 7500 to 9000 tracks. 2409 duplicate plus 1994 imported only totals around 4403 so is somewhat short unless all the other files are kma's.

Check out your Quarantine folder, you may find this in Hoster Data folder, also look in the parent folder where your song files are for a Hoster folder as this may also contain a Quarantined files folder. Or type "Quarantine" into windows search and see if it finds any in relation to Hoster. If you find any kma types in a quarantine folder move them back to their original folder. Any other types other than kma you can just delete.
If you do find any in the quarantine folder and you move them to the kma folder rebuild the database again to include them then merge any duplicates before re-importing all your other kmh types as I suggested before.

[stefbeme]

I have gone back, added the files from quarantine, double checked to make sure there are no other files in the KMA folder as well have checked for duplicates again. I have now successfully added 6000 new files. Hoorah!

My other files that are zipped I am using the index option, it has added appx 40,000 files and now is causing hoster to stop responding. It was working fine for the first 40,000, but I still have about 35,000 to go. I have restarted my computer, restarted hoster and still no success.

Please advise. Thanks!!

[Roy Dennis]

Originally Posted by stefbeme

I have gone back, added the files from quarantine, double checked to make sure there are no other files in the KMA folder as well have checked for duplicates again. I have now successfully added 6000 new files. Hoorah!

My other files that are zipped I am using the index option, it has added appx 40,000 files and now is causing hoster to stop responding. It was working fine for the first 40,000, but I still have about 35,000 to go. I have restarted my computer, restarted hoster and still no success.

Please advise. Thanks!!

How much free space showing on your hard drive ?
When you added the files from the quarantine folder did you copy them or move them ?
If you copied them then you are using twice the space and may be the reason you are getting the not responding, you need to delete anything in the quarantine folders once you have copied them.

[stefbeme]

At the time, I had 101GB, so I deleted the contents of the Quarantine and dumped my recycle bin, now up to 160. I was able to index a few more files and it stops again.
Now I am one of those OCD folks, so I was using the option for DiscID-Track_Artist_title. Thinking this may be causing the problem I am now trying to import the remaining files using the Ignore-Title-Artist option. I have attempted 3 times and it has halted Hoster "not responding", which closes Hoster automatically.

[Roy Dennis]

Is that 160 Gb left or 160 MB ?

Are you importing the the zip files as kma or kmh ?

If you are importing them as kma then you are doubling up on your storage.

If you import them as kma then you no longer need the original zip file either store the zip on another hard drive for backup purposes or delete them, else import them as kmh type which will only add a small header file of a few bytes to each file that remains in the same folder as the zip.