FTP and SFTP items on scheduler fail intermittently

mkrause · 2022-08-31 18:17:51

Hi,
After upgrading to latest 4.20 version (including the prior one to most recent patch, same problem) I am having intermittent problems with scheduled items using either FTP or SFTP datasinks. Both are going to localhost (127.0.0.1) FTP servers, that bind correctly to those ports and respond correctly from shell prompt on that server 100% of the time.

For SFTP, I get this when it does fail (FYI, when running datasink tests directly from datasink area, it is successful 100% of the time):

Error-Details: net.datenwerke.scheduler.service.scheduler.exceptions.ActionExecutionException: report could not be sent to SFTP
at net.datenwerke.rs.ftp.service.ftp.action.ScheduleAsSftpFileAction.sendViaSFTPDatasink(ScheduleAsSftpFileAction.java:155)
at net.datenwerke.rs.ftp.service.ftp.action.ScheduleAsSftpFileAction.execute(ScheduleAsSftpFileAction.java:93)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.executeAction(SchedulerTask.java:630)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.executeActions(SchedulerTask.java:513)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.safeCall(SchedulerTask.java:207)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.call(SchedulerTask.java:99)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.call(SchedulerTask.java:1)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: known_hosts file does not exist
at net.datenwerke.rs.ftp.service.ftp.FtpSenderServiceImpl.configureSftp(FtpSenderServiceImpl.java:143)
at net.datenwerke.rs.ftp.service.ftp.FtpSenderServiceImpl.sendToFtpServer(FtpSenderServiceImpl.java:93)
at net.datenwerke.rs.ftp.service.ftp.SftpServiceImpl.doExportIntoDatasink(SftpServiceImpl.java:47)
at net.datenwerke.rs.core.service.datasinkmanager.DatasinkServiceImpl.exportIntoDatasink(DatasinkServiceImpl.java:94)
at net.datenwerke.rs.ftp.service.ftp.action.ScheduleAsSftpFileAction.sendViaSFTPDatasink(ScheduleAsSftpFileAction.java:140)
... 10 more

And failure via FTP:
Error-Details: net.datenwerke.scheduler.service.scheduler.exceptions.ActionExecutionException: report could not be sent to FTP
at net.datenwerke.rs.ftp.service.ftp.action.ScheduleAsFtpFileAction.sendViaFTPDatasink(ScheduleAsFtpFileAction.java:146)
at net.datenwerke.rs.ftp.service.ftp.action.ScheduleAsFtpFileAction.execute(ScheduleAsFtpFileAction.java:91)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.executeAction(SchedulerTask.java:630)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.executeActions(SchedulerTask.java:513)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.safeCall(SchedulerTask.java:207)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.call(SchedulerTask.java:99)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.call(SchedulerTask.java:1)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: net.datenwerke.rs.core.service.datasinkmanager.exceptions.DatasinkExportException: An error occurred during datasink export
at net.datenwerke.rs.ftp.service.ftp.FtpServiceImpl.doExportIntoDatasink(FtpServiceImpl.java:49)
at net.datenwerke.rs.core.service.datasinkmanager.DatasinkServiceImpl.exportIntoDatasink(DatasinkServiceImpl.java:94)
at net.datenwerke.rs.ftp.service.ftp.action.ScheduleAsFtpFileAction.sendViaFTPDatasink(ScheduleAsFtpFileAction.java:130)
... 10 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not connect to FTP server on "127.0.0.1".
at org.apache.commons.vfs2.provider.ftp.FtpClientFactory$ConnectionFactory.createConnection(FtpClientFactory.java:232)
at org.apache.commons.vfs2.provider.ftp.FtpClientFactory.createConnection(FtpClientFactory.java:272)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.createClient(FTPClientWrapper.java:106)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.createClient(FTPClientWrapper.java:98)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.getFtpClient(FTPClientWrapper.java:147)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.<init>(FTPClientWrapper.java:53)
at org.apache.commons.vfs2.provider.ftp.FtpFileProvider.doCreateFileSystem(FtpFileProvider.java:70)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:92)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:71)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:55)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:788)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:835)
at net.datenwerke.rs.ftp.service.ftp.FtpSenderServiceImpl.sendToFtpServer(FtpSenderServiceImpl.java:103)
at net.datenwerke.rs.ftp.service.ftp.FtpServiceImpl.doExportIntoDatasink(FtpServiceImpl.java:47)
... 12 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:609)
at org.apache.commons.net.SocketClient._connect(SocketClient.java:253)
at org.apache.commons.net.SocketClient.connect(SocketClient.java:212)
at org.apache.commons.vfs2.provider.ftp.FtpClientFactory$ConnectionFactory.createConnection(FtpClientFactory.java:165)
... 25 more

eduardo · 2022-09-01 09:59:05

Hi mkrause,

this is very strange:

mkrause wrote:

when running datasink tests directly from datasink area, it is successful 100% of the time"

as both use exactly the same mechanism and the same methods. Are you completely sure nothing changes in the meanwhile ? You can manually test the scheduler by executing the "execute now" button. If you do this just after you press the "test datasink" button, does it work? Does this manual test stop working eventually ?

The errors you sent are:
1. known_hosts file does not exist
If you don't move the known_hosts from its (OS) location, then this should not happen, as RS just checks if the file is there, nothing more. You can see this here:
https://github.com/infofabrik/reportser … .java#L142

Pls make completely sure the file is still there.

2. Caused by: java.net.ConnectException: Connection refused (Connection refused)
This means your RS cannot reach the host: https://stackoverflow.com/questions/687 … on-refused

This exception means that there is no service listening on the IP/port you are trying to connect to:
You are trying to connect to the wrong IP/Host or port.
You have not started your server.
Your server is not listening for connections.
On Windows servers, the listen backlog queue is full.

Are you completely sure your server is listening for connections in that moment ?

Regards,
Eduardo

mkrause · 2022-09-01 14:50:27

Yes I realize both of these issues do not line up. The known_hosts is specified in /etc/security/misc.cf in the filesystem like this:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<juel>
<expression>
<blacklist>getClass</blacklist>
</expression>
</juel>
<knownHosts>/etc/ssh/known_hosts</knownHosts>
</configuration>

Also the same file is in the tomcat user's home direct /opt/tomcat/.ssh/known_hosts, so there is no problem there. that's why that log message doesn't make sense.

Likewise, the ftp port (21) *ALWAYS* answers at 127.0.0.1 even when simultaneously testing. So again, not lining up.

One thing: both of these scheduled runs are BIRT reports. Is it possible they are timing out (the queries take -- 5-15 seconds to run) and causing havoc?

And yes, datasinks test OK 100% of the time. I could make a screen capture to show you this behavior if you want and send to you personally. let me know if you need that.

eduardo · 2022-09-02 08:07:16

Hi mkrause,

a question: during SFTP "test datasink", do you get a test text file in your SFTP directory? What the "test datasink" does is to create a simple text file, and then send it to the SFTP directory. Do you get one? Pls make sure you delete it previously if you had one so to make sure it is the new one.

Regards,
Eduardo

eduardo · 2022-09-02 08:34:32

Hi mkrause,

in the error list of your scheduler: are all errors of the same job exactly the same or do you have different errors each time/sometimes ?

Regards,
Eduardo

eduardo · 2022-09-02 09:41:13

Hi mkrause,

thanks for the videoscreen. I tested by scheduling SFTP every minute, and also did some manual executions, and I didn't get any error:

I would suggest two different tests (let's first focus on SFTP):

For both tests, pls create a simple report based on the internal reportserver datasource, and use a simple query like this: "select * from RS_USER") so we make sure your specific report do not cause the problems somehow.

1. can you schedule your RS_USER_VAR (as in my screenshot) report every minute and execute some manually? when do you get an error, can you post a screenshot of your execution list? (you can use http://imgbb.com for this)
Also, if you get an error, pls post it here (it is the same as above or do you have a different error? )

2. in order to make sure your specific reportserver configuration does not cause the error, it might be good to test in a completely new reportserver installation. Can you install this: https://reportserver.net/spp/download-r … -installer in some testing environment ? You can just click next next next to install, and this does not interfere with any other (manual) installation you might have. Of course, pls be careful not to install in your PROD environment
In a completely new reportserver installation, if you schedule your report every minute as above : do you get the errors ? can you also pls post a screenshot of this ?

Regards,
Eduardo

eduardo · 2022-09-02 10:04:26

Hi mkrause,

how are you adding your localhost to your known_hosts file?

ssh-keyscan localhost >> known_hosts

would be the correct way in order not to override the values there. In my case I got this error later:

reject HostKey: localhost

but it was just because I overrided the localhost values with another host ( I entered

ssh-keyscan new_host > known_hosts

instead of

ssh-keyscan new_host >> known_hosts

) .

So if you get this error:

reject HostKey: localhost

can you double check your known_hosts file is correct?

less known_hosts

-> should print your localhost

Regards,
Eduardo

mkrause · 2022-09-02 13:38:59

Answered everything in email.

eduardo · 2022-09-05 08:51:03

Hi mkrause,

pls use the forum for communication, as it would maybe help other users having the same problem. For videoscreens you can of course use the email.

Can you pls send me a videoscreen of your SFTP failing and immediately after of you using the "sftp datasink test" on the same datasink? You mention you have this error "known_hosts file does not exist" , so can you pls test your datasink immediately after getting this error ?

Regards,
Eduardo

eduardo · 2022-09-05 11:31:32

Hi mkrause,

also, pls show in the videoscreen that the scheduled job uses the same datasink as the "test datasink" button. Just to make sure there is no error there.

Regards,
Eduardo

mkrause · 2022-09-05 14:08:47

all set.

eduardo · 2022-09-07 10:37:50

Hi mkrause,

in the SFTP videoscreen you sent I see some inconsistent paths. Maybe the problem has to do somehow with these.

You configure your datasink with this path:
/home/reports/reportserver

and your scheduled job with this path:
reportserver

SFTP uses relative paths. In my case I use this:
path datasink configuration:
./Desktop/sftp

path in job:
./Desktop/sftp (so I don't change the default one)

Then it works and the "test datasink" places the sftp testfile here:
/Users/eduardo/Desktop/sftp
which is my home directory.

If I try to place absolute paths in the datasink configuration:
/Users/eduardo/Desktop/sftp
then it sends the sftp testfile to:
/Users/eduardo/Users/eduardo/Desktop/sftp

which is not what we are aiming.

So pls first try to adapt your SFTP datasink to a local path, and make sure that the "datasink test" button sends the sftp testfile to the correct location (in your videoscreen it doesn't). Then, schedule a new report, leave the default configuration that worked with the "datasink test" button, and test this. In my testcases it always works as soon as the "datasink test" button starts to work correctly.

Regards,
Eduardo

mkrause · 2022-09-08 00:06:38

I changed the datasink to relative path (which is what the job was actually set up as already). Will let you know. The test file now does go to the right place, fyi.

Last edited by mkrause (2022-09-08 00:07:00)

mkrause · 2022-09-08 02:50:54

Same intermittent failure just happened, even now both datasink and scheduled item using a relative path.

eduardo · 2022-09-09 08:44:32

Hi mkrause,

we cannot reproduce this, it works without failing in our testcases, so maybe your specific configuration/installation is somehow wrong.
Can you pls test with a clean, new reportserver, as I suggested here: ?

2. in order to make sure your specific reportserver configuration / installation does not cause the error, it might be good to test in a completely new reportserver installation. Can you install the windows installer package https://reportserver.net/en/download/ in some testing environment ? You can just click next next next to install, and this does not interfere with any other (manual) installation you might have. Of course, pls be careful not to install in your PROD environment
In a completely new reportserver installation, if you schedule your report every minute as above : do you get the errors ? can you also pls post a screenshot of this ?

Regards,
Eduardo

mkrause · 2022-09-09 15:53:08

Question: in the event a new clean environment solves the problem, my big pain point then becomes migrating all of the reports (many), users, and permissions, I have already. Is there a way to do this? I can also selectively copy over specific SQL tables to get this done. Looking for info on this.

Felix · 2022-09-16 06:51:57

Hi mkrause!

My appologies for the late answer, maybe this post is helpful?

As Eduardo mentioned, in order to migrate content from one reportserver to another, you have to set up a reportserver with the exact same version, import your content and THEN performe a upgrade.

Perhaps this tutorial might help you as well on how to import a Report.

While developing I am frequently using the "export" button in the administration view (at least for Files, Dashboards and Reports). There you can select the Report Root, Export (as File or as Text... I am using text to have it in the clipboard ) and then paste this text in the "Import data" field in Administration => Import => Start import (button) => Import data (text area in a dialog that should open)

Regards,
Felix

mkrause · 2022-09-20 17:22:05

I am experimenting with a fresh Windows based install (using Tomcat 9.065 and Java 11), and manually moving over reports .. (versus the ubuntu install I normally use) to get this SFTP working more reliably.

And I am successful getting the reportserver up and running. However, I am not so successful with setting up the new datasink.

On the new windows machine test for the SFTP datasink, I am getting a "known_hosts file does not exist" error when testing the datasink on the windows setup. Now the filesystem structure and user structure is all very different on the windows vs unix environment. On this test machine, I set up /etc/security/misc.cf to have this configuration (and populated a known_hosts file):

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<juel>
<expression>
<blacklist>getClass</blacklist>
</expression>
</juel>
<errorMessages>
<hideViolatedSecurityExceptionDetails>false</hideViolatedSecurityExceptionDetails>
</errorMessages>
<knownHosts>C:\known_hosts</knownHosts>
<disableLostPassword>false</disableLostPassword>
</configuration>

But that made no difference. So almost there. Please advise? How do get rid of the no known_hosts error in a windows environment?

mkrause · 2022-09-30 20:44:05

Never heard back to prior questions, but thought I'd update the forum: when switching to an SFTP account that uses a private key as a login requirement, i have had zero failures all week. So something tells me that code path does not present the opportunity for this bug to surface.

Re above: I did not pursue the windows install because of the issues above with known_hosts in that installation (do not know how to solve), but this is kind of moot.

ReportServer Forum

#1 2022-08-31 18:17:51

FTP and SFTP items on scheduler fail intermittently

#2 2022-09-01 09:59:05

Re: FTP and SFTP items on scheduler fail intermittently

#3 2022-09-01 14:50:27

Re: FTP and SFTP items on scheduler fail intermittently

#4 2022-09-02 08:07:16

Re: FTP and SFTP items on scheduler fail intermittently

#5 2022-09-02 08:34:32

Re: FTP and SFTP items on scheduler fail intermittently

#6 2022-09-02 09:41:13

Re: FTP and SFTP items on scheduler fail intermittently

#7 2022-09-02 10:04:26

Re: FTP and SFTP items on scheduler fail intermittently

#8 2022-09-02 13:38:59

Re: FTP and SFTP items on scheduler fail intermittently

#9 2022-09-05 08:51:03

Re: FTP and SFTP items on scheduler fail intermittently

#10 2022-09-05 11:31:32

Re: FTP and SFTP items on scheduler fail intermittently

#11 2022-09-05 14:08:47

Re: FTP and SFTP items on scheduler fail intermittently

#12 2022-09-07 10:37:50

Re: FTP and SFTP items on scheduler fail intermittently

#13 2022-09-08 00:06:38

Re: FTP and SFTP items on scheduler fail intermittently

#14 2022-09-08 02:50:54

Re: FTP and SFTP items on scheduler fail intermittently

#15 2022-09-09 08:44:32

Re: FTP and SFTP items on scheduler fail intermittently

#16 2022-09-09 15:53:08

Re: FTP and SFTP items on scheduler fail intermittently

#17 2022-09-16 06:51:57

Re: FTP and SFTP items on scheduler fail intermittently

#18 2022-09-20 17:22:05

Re: FTP and SFTP items on scheduler fail intermittently

#19 2022-09-30 20:44:05

Re: FTP and SFTP items on scheduler fail intermittently

Board footer