[C++] Recursive S/FTP Sync - Guidance Needed

[BinaryData]

I've got a project I'm working on, and I need some advice on how to code something. Let me explain the goal.

 

I have 3 servers that my team uploads files too every day. I have a backup server that I run. It's tedious as all hell to download from each one, every day. I don't know what is new, and what isn't. The 3 Servers run CentOS, while my server runs Windows. Here's what I need it to do.

 

Every 24 hours, I want it to check for new files, if there are new files, it initiates a download to my back up server. There is a specific time frame that the server will be able to download. I have a relatively fast internet, however the server is only running a dual core with 15TB of Storage in a Pooled Environment. The download can run from 12AM PST to 3PM PST, a 15 hour window. At most that would be uploaded/downloaded would be 5GB or less, depending on whats being updated. My server at home will be running sftp, so there isn't that problem. I could make the files download in ftps, and only accept connections from my 3 servers.

 

I've never worked on anything in Linux, it's always been Windows. I'm open to suggestions for other software for syncing, but I'd like to build this on my own.

[Circaflex]

@BudMan, @Jaredor @sc302 would probably know the best and most efficient way. Tagging them as I don't know if they browse this subforum often.

[BinaryData]

17 hours ago, Circaflex said:

@BudMan, @Jaredor @sc302 would probably know the best and most efficient way. Tagging them as I don't know if they browse this subforum often.

I try and not stick my hand in that particular honey pot too often. I know their more than willing to lend their expertise, but I consult them way too much, haha.

 

I know there are excellent FTP Libraries. I'm wondering if the same libraries will work in Linux too.

[simplezz]

If you're talking about transferring files via SSH's FTP, then it wouldn't be too difficult. In fact you could probably do it with a simple script (sftp is supplied with OpenSSH) rather than a fully fledged C++ program. Either way, you'd likely want to run a scheduled Cron job to perform the task.

 

There are plenty of libraries to plug into if you want to do it in code.

[BinaryData]

5 minutes ago, simplezz said:

If you're talking about transferring files via SSH's FTP, then it wouldn't be too difficult. In fact you could probably do it with a simple script (sftp is supplied with OpenSSH) rather than a fully fledged C++ program. Either way, you'd likely want to run a scheduled Cron job to perform the task.

 

There are plenty of libraries to plug into if you want to do it in code.

I was looking at libssh2, actually. I don't have a good understanding of ssh, so I thought this might be a good opportunity to learn about it. I also don't know much about Linux and how things are coded there. Which interests me quite a bit. I came up with a plan on how it should work, coding it will be a bit more difficult. I'm not sure if you can code a Linux Application in Visual Studio and then compile and export it to Windows. I was looking into communication, i.e.

 

Server 1 - Begins the Transfers to Server X, Server 2 - Waits until Server 1 completes, and Server 3 - Waits until Server 2 is complete. So on and so forth. I think that might be the hardest part.

[simplezz]

1 minute ago, BinaryData said:

I was looking at libssh2, actually. I don't have a good understanding of ssh, so I thought this might be a good opportunity to learn about it. I also don't know much about Linux and how things are coded there. Which interests me quite a bit. I came up with a plan on how it should work, coding it will be a bit more difficult. I'm not sure if you can code a Linux Application in Visual Studio and then compile and export it to Windows. I was looking into communication, i.e.

 

Server 1 - Begins the Transfers to Server X, Server 2 - Waits until Server 1 completes, and Server 3 - Waits until Server 2 is complete. So on and so forth. I think that might be the hardest part.

libssh is a good start. curl is another possibility, though I hear it's a pita to work against when dealing with authentication etc.

 

I suppose it depends upon whether you want to do the whole end-to-end transfer yourself using a client/server architecture, or partially. That is to say, write a client on Linux that does the transfer to your main server that's running a standard sftp server. There's a lot of prebuilt software already out there.

 

[BinaryData]

I'm hoping to not have to code it, but it would be a good project. My colleague pointed out "What about data validation?" I haven't even thought of that portion until he mentioned it. After he mentioned that, I thought about it. I would generate the original crc after the files transferred and then send it, if it matches, we're good. If it doesn't match, request that file again.

 

I'll look into those prebuilt software, I was trying to find a possible way to automate it on CentOS without killing the system resources.