darkmiasma
Trakanon Raider
- 822
- 585
I've got the Synchronized Folders set up, but is there any way to set it to do Segmented Downloading automatically?
Torrents
- Thread starter Malt_sl
- Start date
Void
Experiencer
- 9,447
- 11,122
Not that I'm aware of, but what I did was after the task is listed in the bottom window, I believe I did something like right-clicking on it (might have to be in the particular box, I can't remember) and choosing the HIGH-8 or whatever it was (I'm assuming you followed the Whatbox wiki to set up CuteFTP, which will then give you that HIGH-8 option). You can't ever set up the task with it, but once the task is there you can select it, and it will keep that option. I've had issues with the task not saving properly and having to re-do it, but you can just manually select the higher speed each time you have to do it.
If you can't see what I'm talking about, I will give you the specifics when I get home.
darkmiasma
Trakanon Raider
- 822
- 585
That's exactly what I ended up doing - I was just hoping I missed an option somewhere to have that selected for good.
Edit: had to screw around with the Transfer - Smart Overwrite options, but I think I have the remote to local settings working now.
I have:
Matching Files: Skip (Ignore 0 byte files)
Matching Folders: Use Rules
Rules:
If destination date is <the same> & size is <smaller> during <download>, then <overwrite>.
If destination date is <older> & size is <the same> during <download>, then <skip>.
Prompt timeout: Skip
Sound right? I just want it to mirror the Whatbox folder to my NAS, if it doesn't exist, download it. If it does, but it isn't the same size, re-download it; if it's already there, and the same size, leave it alone.
darkmiasma
Trakanon Raider
- 822
- 585
Well I was way off on those rules - this has been working right all morning though:
Matching Files: Use Rules (Ignore 0 byte files)
Matching Folders: Use Rules
Rules:
If destination date is <no matter> & size is <smaller> during <download>, then <resume>.
Prompt timeout: Resume
Matching Files: Use Rules (Ignore 0 byte files)
Matching Folders: Use Rules
Rules:
If destination date is <no matter> & size is <smaller> during <download>, then <resume>.
Prompt timeout: Resume
Void
Experiencer
- 9,447
- 11,122
I didn't notice your edit before, so I didn't reply, sorry about that.
What I've done that seems to work best for me is just having everything download on the Whatbox to an "In Progress" folder, and when it finishes it is moved to a "Completed" folder. I only mirror the "Completed" one, obviously. Then I tell it that if it finds a matching file, just skip it. I'm sure I could do like you've done there to be more complete, but I basically copy everything out of my home computer folder into various permanent folders, so I just make sure it is not zero bytes. If it is, I delete it from home and let it re-download next round.
Your solution is probably a better one, but I would still use a Completed folder if you aren't, to avoid as many conflicts as possible. I will probably modify mine to be more like yours all the same. Let me know if you have any issues. I've only had one, and I'm guessing it was that one in a million time when something was moving to the Completed folder just as CuteFTP checked the directory. Should be pretty rare, but obviously something happened all the same.
What I've done that seems to work best for me is just having everything download on the Whatbox to an "In Progress" folder, and when it finishes it is moved to a "Completed" folder. I only mirror the "Completed" one, obviously. Then I tell it that if it finds a matching file, just skip it. I'm sure I could do like you've done there to be more complete, but I basically copy everything out of my home computer folder into various permanent folders, so I just make sure it is not zero bytes. If it is, I delete it from home and let it re-download next round.
Your solution is probably a better one, but I would still use a Completed folder if you aren't, to avoid as many conflicts as possible. I will probably modify mine to be more like yours all the same. Let me know if you have any issues. I've only had one, and I'm guessing it was that one in a million time when something was moving to the Completed folder just as CuteFTP checked the directory. Should be pretty rare, but obviously something happened all the same.
darkmiasma
Trakanon Raider
- 822
- 585
Yup - I have the auto-move enabled to a separate folder, and that's the folder I set up in CuteFTP to sync using the last round of settings I posted - it works perfectly.
Void
Experiencer
- 9,447
- 11,122
Jesus fuck, I put in a support ticket to change my plan to one of the new FlexNL plans, and literally less than a minute later I realized I wanted to move up to the $25 plan instead of the $15 one, because it offers just a fuckload more disk space, and they had already replied saying the transfer would start shortly. I'm not kidding it said something like "1 min ago" on my original ticket, and "42 seconds ago" on the response. Talk about quick fucking customer service, Jesus.
I was able to upgrade to the higher one, but wow, I am not even going to bother ever looking for a lower price or anything if their support remains at this level.
Once I get the transfer done I'll mess with trying to figure out Dropbox. For people that might wonder why I would prefer that to just right-clicking in the browser, it is precisely because of the stuff darkmiasma and I were talking about in the previous posts. If I manually add a torrent with the right-click, I have to put it into the "In Progress" folder, and then when it finishes manually move it to the "Completed" folder so that CuteFTP will download it to my home computer. If you have a watch folder in Dropbox, however, it will do all of that automatically. In only a month I've forgotten to do the manual transferring several times from work, got home and realized I had to wait for the damn shows to download, and lamented my lack of Dropbox awesomeness. So that's why I prefer that method.
I was able to upgrade to the higher one, but wow, I am not even going to bother ever looking for a lower price or anything if their support remains at this level.
Once I get the transfer done I'll mess with trying to figure out Dropbox. For people that might wonder why I would prefer that to just right-clicking in the browser, it is precisely because of the stuff darkmiasma and I were talking about in the previous posts. If I manually add a torrent with the right-click, I have to put it into the "In Progress" folder, and then when it finishes manually move it to the "Completed" folder so that CuteFTP will download it to my home computer. If you have a watch folder in Dropbox, however, it will do all of that automatically. In only a month I've forgotten to do the manual transferring several times from work, got home and realized I had to wait for the damn shows to download, and lamented my lack of Dropbox awesomeness. So that's why I prefer that method.
Void
Experiencer
- 9,447
- 11,122
Transfer went smoothly and with no hitch, other than obviously changing my IP address, and my CuteFTP at home won't be connected until I change it, but that's to be expected. Those guys are awesome.
For those looking for new torrent sites, webop.me is apparently open to new registrations right now. They are porn only, and have a pretty decent selection. IPT is better overall, and newsgroups and buying shit like a rapidgator premium account to go with some forums seem to be even more useful, but if you want some free porn in a place that is pretty easy to keep ratio with, just in case IPT ever disappears, then I'd suggest giving Webop a try.
For those looking for new torrent sites, webop.me is apparently open to new registrations right now. They are porn only, and have a pretty decent selection. IPT is better overall, and newsgroups and buying shit like a rapidgator premium account to go with some forums seem to be even more useful, but if you want some free porn in a place that is pretty easy to keep ratio with, just in case IPT ever disappears, then I'd suggest giving Webop a try.
Void
Experiencer
- 9,447
- 11,122
Messing around with Dropbox on the seedbox, and I have it all figured out for the most part I think, but I can't figure out a way to exclude ALL of my dropbox at once, and then just remove the exclusion from one specific folder. I've got dozens of folders within my main Dropbox folder, and I can't imagine typing every single one of them in to exclude. I googled and nothing seems to indicate an all inclusive exclusion. Any ideas? Or can I just use *.* or something similar?
darkmiasma
Trakanon Raider
- 822
- 585
Unfortunately you have to list all the directories - I considered throwing all of my directories into a single folder, just so I could exclude that (since it does subfolders automatically). In the end, I just typed them all out, since you can hit the up arrow in putty and it puts the last command in there, so I just changed the directory name real fast everytime.
Void
Experiencer
- 9,447
- 11,122
Actually, I just did ~/dropbox.py exclude add ~/Dropbox/* and it seems to have worked. Now to see if it starts syncing any new folders I make, since they aren't specifically excluded.
It has also indicated to me that I need to clean that folder up!
Thanks for the help, appreciate it.
It has also indicated to me that I need to clean that folder up!
Thanks for the help, appreciate it.
darkmiasma
Trakanon Raider
- 822
- 585
Excellent - I didn't even think to try ~/Dropbox/*
I made an empty directory to test it, and it did sync it to my Whatbox, so you'd have to go back and exclude new top level directories from here on out. With that in mind, I might make a single directory and put everything in there, and never have to worry about it again after the initial exclusion (I tested a new folder in an already excluded one, and it didn't pop up). Might be the best practice do just do it that way, so we never have to remember to make new exclusions.
Yeah, thanks for your help as well, and everyone else in the thread who put me onto this.
Edit: tomorrow I'll post a step-by-step writeup for people who are interested in Whatbox/Dropbox/CuteFTP to set everything up for loading .torrent files with Dropbox, moving completed files automatically, and setting up the folder sync with CuteFTP so that it auto-downloads the completed files, and resumes any broken/unfinished files.
I made an empty directory to test it, and it did sync it to my Whatbox, so you'd have to go back and exclude new top level directories from here on out. With that in mind, I might make a single directory and put everything in there, and never have to worry about it again after the initial exclusion (I tested a new folder in an already excluded one, and it didn't pop up). Might be the best practice do just do it that way, so we never have to remember to make new exclusions.
Yeah, thanks for your help as well, and everyone else in the thread who put me onto this.
Edit: tomorrow I'll post a step-by-step writeup for people who are interested in Whatbox/Dropbox/CuteFTP to set everything up for loading .torrent files with Dropbox, moving completed files automatically, and setting up the folder sync with CuteFTP so that it auto-downloads the completed files, and resumes any broken/unfinished files.
galgor
Bronze Knight of the Realm
- 128
- 16
Yes they do.
You guys know that they developed their own browser-based file-browser called Helm, right? You can drop files right into your 'watch' folder using it. No need for fooling around with dropbox... at least for uploading multiple torrents.
Void
Experiencer
- 9,447
- 11,122
Yeah, I have used Helm multiple times, most recently to clean up the stuff Dropbox started downloading before I figured out how to exclude everything. It is pretty nice, much better than a lot of others I've seen. But you would still have to save the torrent file to your desktop or something and then drag it across. Once Dropbox is set up, it doesn't matter if I'm at a computer with the Whatbox management tools or not, I just have to have Dropbox. Which I do on every computer, phone, and tablet I own. Admittedly it was a little confusing to set up at first, but now that it is, I'll never have to worry about it again, and I like it better. And since darkmiasma is going to post a step-by-step instruction manual for getting it all going, it won't be difficult for new people to get to where we are either.
darkmiasma
Trakanon Raider
- 822
- 585
OK, here is the guide for setting up your Whatbox, Dropbox Syncing for uploading .torrents from anywhere, and FTP Segmented Downloading back to your NAS/PC ... spoilered for size.
Let me know if I missed anything.
Let me know if I missed anything.
What you'll need:
Whatbox.ca Account/Seedbox
CuteFTP (for segmented downloads)
Dropbox (enable .torrent file pickups)
Putty (SSH to setup Seedbox options)
ruTorrent (installed by default)
----------
Before we start, make sure when following the directions to substitute the name of your server instead of the example of nacho.
----------
Create a "Completed" directory:
Log into your home directory (eg.https://nacho.whatbox.ca/filebrowser)
Click on New Folder and name it something (eg. -Completed-)
Easy enough, now let's move on.
-----------
Install/Setup Dropbox:
- Use Putty and SSH into your server. (eg. hostname: nacho.whatbox.ca)
- When prompted put your Whatbox username in and hit enter.
- When prompted put your Whatbox password in and hit enter.
- Make sure that you are in the root of your home directory with cd
- Since all Whatbox servers have an x86-64 architecture, we need to download the x86-64 Dropbox client: wget -O dropbox.tar.gz "http://www.dropbox.com/download/?plat=lnx.x86_64"
- Extract the newly download archive: tar -xvzf dropbox.tar.gz
- Run the Dropbox daemon: ~/.dropbox-dist/dropboxd
You should now see output similar to this:
This client is not linked to any account... Please visithttps://www.dropbox.com/cli_link?hos...da0x67334d02c1to link this machine.
Do not close the process until you finish logging in and linking your account on the Dropbox website. dropboxd will create a ~/Dropbox folder and start synchronizing it after this step! Go to the URL given in your SSH client; you should see a success message at the top of your screen.
Once this is finished, we can move on to installing the Dropbox CLI:
- Download the Python script: wget -O ~/dropbox.py "http://www.dropbox.com/download?dl=packages/dropbox.py"
- Give your account access to execute the script: chmod 755 ~/dropbox.py
- Make sure Dropbox is running: ~/dropbox.py start
Now we're going to exclude all of your Dropbox directories from syncing with the server, and then create a directory where we'll drop new torrents into:
Run this command to exclude everything: ~/dropbox.py exclude add ~/Dropbox/*
Now in your normal Dropbox app, or from their website create a new folder for the .torrent files (eg. torrents). Since we created this after we excluded everything, this will be the only folder that is now synced to your Whatbox. Now whenever you download any .torrent files, place them in this folder, and the server will automatically start downloading them (it scans every 5 minutes by default).
----------
Setting up ruTorrent:
Log into your WebUI (eg. ruTorrent:https://nacho.whatbox.ca/webui)
Click on Settings (the Blue Gear at the top of the page):
- Click on Autotools
- Enable "AutoMove" feature
- Click "..." to browse for the -Completed- directory we set up earlier
- Operation type: Move
- Enable "AutoWatch" feature
- Click "..." to browse for the torrents directory we set up earlier
- Check start download automatically.
- Click OK.
----------
Setting up CuteFTP to sync your completed items back to your NAS/PC:
- Fill in the info for your site (eg. host: nacho.whatbox.ca) (I personally used SFTP using SSH2 for my protocol type)
- To set up segmented downloading either do the following or follow this link (if you like pictures -only do steps 1-4):https://whatbox.ca/wiki/CuteFTP_Pro#...ed_downloading
- Right click on your site (assuming you made it) and select properties
- Click on the Options tab, and select "Use site specific option"
- Change the site max concurrent transfers to 8. Hit OK to exit.
- Go to Tools - Global Options.
- Go to the Connection tab
- Change Global max concurrent transfers to 50.
- Change Per site max concurrent transfers to 20.
- Go to the Transfer tab
Change When selecting MAX in a multi-part transfer, use __ to 8
- Hit OK to exit.
- Back in the main windows Click on Tools - Folder Tools - Synchronize Folders.
- In the Enter the local path to synchronize box, click the folder icon to browse for the folder.
- In the Enter the remote path to synchronize box, type the remote path of the folder.
- To include the subfolders within the folder you want to synchronize, select the Include subfolders check box.
- Click Next.
- In the Select the direction of synchronization box, click the following option: Mirror Remote keeps the remote folder the same and changes the local folder to match the remote folder.
- Click Next.
- In the For matching filenames always box, click the following: Use global overwrite settings. This is the default selection. This option uses overwrite settings configured for all sites in Tools > Global Options > Transfer > Smart Overwrite.
- In the Remote file names that differ only in case box, if the file names are identically spelled but differently cased, you can select the following: Transfer first file only and skip others. The first file transfers and the rest are skipped.
- To make the mirrored folder a strict duplicate of the source folder, select the When one way mirroring, delete the destination file if it does not exist in the source folder check box. This updates old files and deletes unused files. If you clear this check box, the extra files present in the mirrored folder are left in place. I personally clear it, as I might not have moved files out of my local NAS sync folder before I cleared files from the -Completed- folder on my Whatbox.
- To prompt you before deleting the file, select the Always prompt before deleting check box.
- Click Next.
- To prevent CuteFTP from following symbolic links when mirroring subfolders, select the Ignore symbolic link check box. Select this option unless symbolic links are absolutely needed. If you clear this check box and a symbolic link points to a spot further up in the folder tree from where the mirroring began, you are stuck in an infinite loop.
- To compensate for a problem that occurs when trying to synchronize ASCII files residing on UNIX file systems, select the Transfer all files in binary to compensate for ASCII file differences check box. Windows and UNIX differ slightly on how they store ASCII files, especially with regard to the End of Line (EOL) or line breaks in ASCII documents. This difference causes an otherwise identical file to be seen as "different" when performing a synchronization tasks because of size difference. All files are transferred in binary mode, which means that an ASCII file residing on a local system is transferred to the remote server with its EOLs intact.
- Click Next.
- Click the following: To start synchronization of folders at a specified time, click Schedule.
- Click Next.
- Do the following:
- In the Start on area, specify the date and time for the synchronization.
- To repeat the synchronization after a specified period, select the Enable recurrence check box, then in the Repeat every box, specify the frequency. (I use every 30 minutes)
- Click Next. The synchronization settings appear.
- Verify the synchronization settings, then click Finish to save the settings or click back to edit the settings.
The synchronization task appears in the Queue Window.
- Right-click on your task and choose Multi-part Values - MAX (HIGH-8). You can't ever set up the task with it, but once the task is there you can select it, and it will keep that option.
Almost done ... couple of things:
Go back to Tools - Global Options:
- Go to Transfer - Smart Overwrite (use the following settings):
Matching Files: Use Rules (Ignore 0 byte files)
Matching Folders: Use Rules
Rules:
If destination date is <no matter> & size is <smaller> during <download>, then <resume>.
Prompt timeout: Resume
- Hit OK to exit.
You're all done.
Whatbox.ca Account/Seedbox
CuteFTP (for segmented downloads)
Dropbox (enable .torrent file pickups)
Putty (SSH to setup Seedbox options)
ruTorrent (installed by default)
----------
Before we start, make sure when following the directions to substitute the name of your server instead of the example of nacho.
----------
Create a "Completed" directory:
Log into your home directory (eg.https://nacho.whatbox.ca/filebrowser)
Click on New Folder and name it something (eg. -Completed-)
Easy enough, now let's move on.
-----------
Install/Setup Dropbox:
- Use Putty and SSH into your server. (eg. hostname: nacho.whatbox.ca)
- When prompted put your Whatbox username in and hit enter.
- When prompted put your Whatbox password in and hit enter.
- Make sure that you are in the root of your home directory with cd
- Since all Whatbox servers have an x86-64 architecture, we need to download the x86-64 Dropbox client: wget -O dropbox.tar.gz "http://www.dropbox.com/download/?plat=lnx.x86_64"
- Extract the newly download archive: tar -xvzf dropbox.tar.gz
- Run the Dropbox daemon: ~/.dropbox-dist/dropboxd
You should now see output similar to this:
This client is not linked to any account... Please visithttps://www.dropbox.com/cli_link?hos...da0x67334d02c1to link this machine.
Do not close the process until you finish logging in and linking your account on the Dropbox website. dropboxd will create a ~/Dropbox folder and start synchronizing it after this step! Go to the URL given in your SSH client; you should see a success message at the top of your screen.
Once this is finished, we can move on to installing the Dropbox CLI:
- Download the Python script: wget -O ~/dropbox.py "http://www.dropbox.com/download?dl=packages/dropbox.py"
- Give your account access to execute the script: chmod 755 ~/dropbox.py
- Make sure Dropbox is running: ~/dropbox.py start
Now we're going to exclude all of your Dropbox directories from syncing with the server, and then create a directory where we'll drop new torrents into:
Run this command to exclude everything: ~/dropbox.py exclude add ~/Dropbox/*
Now in your normal Dropbox app, or from their website create a new folder for the .torrent files (eg. torrents). Since we created this after we excluded everything, this will be the only folder that is now synced to your Whatbox. Now whenever you download any .torrent files, place them in this folder, and the server will automatically start downloading them (it scans every 5 minutes by default).
----------
Setting up ruTorrent:
Log into your WebUI (eg. ruTorrent:https://nacho.whatbox.ca/webui)
Click on Settings (the Blue Gear at the top of the page):
- Click on Autotools
- Enable "AutoMove" feature
- Click "..." to browse for the -Completed- directory we set up earlier
- Operation type: Move
- Enable "AutoWatch" feature
- Click "..." to browse for the torrents directory we set up earlier
- Check start download automatically.
- Click OK.
----------
Setting up CuteFTP to sync your completed items back to your NAS/PC:
- Fill in the info for your site (eg. host: nacho.whatbox.ca) (I personally used SFTP using SSH2 for my protocol type)
- To set up segmented downloading either do the following or follow this link (if you like pictures -only do steps 1-4):https://whatbox.ca/wiki/CuteFTP_Pro#...ed_downloading
- Right click on your site (assuming you made it) and select properties
- Click on the Options tab, and select "Use site specific option"
- Change the site max concurrent transfers to 8. Hit OK to exit.
- Go to Tools - Global Options.
- Go to the Connection tab
- Change Global max concurrent transfers to 50.
- Change Per site max concurrent transfers to 20.
- Go to the Transfer tab
Change When selecting MAX in a multi-part transfer, use __ to 8
- Hit OK to exit.
- Back in the main windows Click on Tools - Folder Tools - Synchronize Folders.
- In the Enter the local path to synchronize box, click the folder icon to browse for the folder.
- In the Enter the remote path to synchronize box, type the remote path of the folder.
- To include the subfolders within the folder you want to synchronize, select the Include subfolders check box.
- Click Next.
- In the Select the direction of synchronization box, click the following option: Mirror Remote keeps the remote folder the same and changes the local folder to match the remote folder.
- Click Next.
- In the For matching filenames always box, click the following: Use global overwrite settings. This is the default selection. This option uses overwrite settings configured for all sites in Tools > Global Options > Transfer > Smart Overwrite.
- In the Remote file names that differ only in case box, if the file names are identically spelled but differently cased, you can select the following: Transfer first file only and skip others. The first file transfers and the rest are skipped.
- To make the mirrored folder a strict duplicate of the source folder, select the When one way mirroring, delete the destination file if it does not exist in the source folder check box. This updates old files and deletes unused files. If you clear this check box, the extra files present in the mirrored folder are left in place. I personally clear it, as I might not have moved files out of my local NAS sync folder before I cleared files from the -Completed- folder on my Whatbox.
- To prompt you before deleting the file, select the Always prompt before deleting check box.
- Click Next.
- To prevent CuteFTP from following symbolic links when mirroring subfolders, select the Ignore symbolic link check box. Select this option unless symbolic links are absolutely needed. If you clear this check box and a symbolic link points to a spot further up in the folder tree from where the mirroring began, you are stuck in an infinite loop.
- To compensate for a problem that occurs when trying to synchronize ASCII files residing on UNIX file systems, select the Transfer all files in binary to compensate for ASCII file differences check box. Windows and UNIX differ slightly on how they store ASCII files, especially with regard to the End of Line (EOL) or line breaks in ASCII documents. This difference causes an otherwise identical file to be seen as "different" when performing a synchronization tasks because of size difference. All files are transferred in binary mode, which means that an ASCII file residing on a local system is transferred to the remote server with its EOLs intact.
- Click Next.
- Click the following: To start synchronization of folders at a specified time, click Schedule.
- Click Next.
- Do the following:
- In the Start on area, specify the date and time for the synchronization.
- To repeat the synchronization after a specified period, select the Enable recurrence check box, then in the Repeat every box, specify the frequency. (I use every 30 minutes)
- Click Next. The synchronization settings appear.
- Verify the synchronization settings, then click Finish to save the settings or click back to edit the settings.
The synchronization task appears in the Queue Window.
- Right-click on your task and choose Multi-part Values - MAX (HIGH-8). You can't ever set up the task with it, but once the task is there you can select it, and it will keep that option.
Almost done ... couple of things:
Go back to Tools - Global Options:
- Go to Transfer - Smart Overwrite (use the following settings):
Matching Files: Use Rules (Ignore 0 byte files)
Matching Folders: Use Rules
Rules:
If destination date is <no matter> & size is <smaller> during <download>, then <resume>.
Prompt timeout: Resume
- Hit OK to exit.
You're all done.
I've been given a bunch of invites to beathau5, a tracker that specialises in electronic music. They use the same tracker software as what.cd and the site seems pretty decent overall. If anyone wants an invite, send me a pm with your email address
Void
Experiencer
- 9,447
- 11,122
Looked it over, and it seems to have everything you'd need, except for possibly the part about using SSH to get into your server. That can easily be found on the Whatbox wiki though, so if you're following along and get to that point and don't know what you're doing, just follow that wiki. Other than that, it is pretty much exactly what we've done.
Nice job, looks great.
darkmiasma
Trakanon Raider
- 822
- 585
Thanks - I just tried to make the SSH instructions more detailed ... there really wasn't much to add, but hopefully that did it.
Anyone have an invite to a good private textbook tracker? A couple of friends asked me to look for some things and none of the public trackers or IRC bots are providing.
