You are not logged in.
- Topics: Active | Unanswered
Pages: 1
#1 2019-05-28 14:52:31
- int-mny
- Member
- Registered: 2019-05-28
sFTP implementation scaling issues
Hello,
we are running the 3.0.7 version of the ReportServer and are using its sFTP capabilities.
Recently, we upgraded to the 3.0.5, because we noticed that the sFTP service would 'hang' after a while. After making some thread-dumps, a database connection leak seemed a likely candidate to cause this and so we upgrade to 3.0.7 because we found that that was fixed in the 3.0.6.
After some initial testing, we see that at least the sFTP service does not hang anymore because of a lack of database connections, but we do see another thing happening: OutOfMemoryErrors for simple things like doing a file listing.
Some context:
* The reports that we are serving are quite big (read: few hundred Mbs max, with a few of those in the same virtual directory).
* ReportServer CE 3.0.7
* Heap size: 16 Gb
After looking at the code for a while, it seems that the code is reading all the content of the files so that it can generate a file listing. The below is what I have found so far.
Each number on the left-hand side is a point where a part of the stack is walked, starting at (6) and ending at (0). The (0) signifies the point where the complete contents of a file is read from the database, so that the file size can be shown to the user to implement an 'ls -al'.
----
File: RsSftpSubSystem
protected void sendName(int id, Iterator<SshFile> files) throws IOException {
Buffer buffer = new Buffer();
buffer.putByte((byte) SSH_FXP_NAME);
buffer.putInt(id);
int wpos = buffer.wpos();
buffer.putInt(0);
int nb = 0;
while (files.hasNext() && buffer.wpos() < MAX_PACKET_LENGTH) {
SshFile f = files.next();
buffer.putString(f.getName());
if (version <= 3) {
(6) buffer.putString(getLongName(f)); // Format specified in the specs
} else {
buffer.putString(f.getName()); // Supposed to be UTF-8
}
writeAttrs(buffer, f);
nb++;
}
int oldpos = buffer.wpos();
buffer.wpos(wpos);
buffer.putInt(nb);
buffer.wpos(oldpos);
send(buffer);
}
private String getLongName(SshFile f) {
String username = f.getOwner();
if (username.length() > 8) {
username = username.substring(0, 8);
} else {
for (int i = username.length(); i < 8; i++) {
username = username + " ";
}
}
(5) long length = f.getSize();
String lengthString = String.format("%1$8s", length);
StringBuilder sb = new StringBuilder();
sb.append((f.isDirectory() ? "d" : "-"));
sb.append((f.isReadable() ? "r" : "-"));
sb.append((f.isWritable() ? "w" : "-"));
sb.append((f.isExecutable() ? "x" : "-"));
sb.append((f.isReadable() ? "r" : "-"));
sb.append((f.isWritable() ? "w" : "-"));
sb.append((f.isExecutable() ? "x" : "-"));
sb.append((f.isReadable() ? "r" : "-"));
sb.append((f.isWritable() ? "w" : "-"));
sb.append((f.isExecutable() ? "x" : "-"));
sb.append(" ");
sb.append(" 1");
sb.append(" ");
sb.append(username);
sb.append(" ");
sb.append(username);
sb.append(" ");
sb.append(lengthString);
sb.append(" ");
sb.append(getUnixDate(f.getLastModified()));
sb.append(" ");
sb.append(f.getName());
return sb.toString();
}
----
File: SshFileWrapper
protected Long doGetSize() {
(4) return location.getSize();
}
----
File: VFSLocation
public long getSize() {
if(cannotExist || isRoot() || isFileSystemRoot())
return 0;
if(isWildcardLocation())
return 0;
if(isVirtualLocation())
(3) return getVirtualContentProvider().getSize(this);
return getFilesystemManager().getSize(this);
}
----
File: VirtualContentProviderImpl
@Override
public long getSize(VFSLocation vfsLocation) {
VFSLocation baseLocation = vfsLocation.getVirtualBaseLocation();
if(vfsLocation.equals(baseLocation))
return 0;
(2) return doGetSize(vfsLocation);
}
protected long doGetSize(VFSLocation vfsLocation) {
try {
(0) byte[] content = doGetContent(vfsLocation);
return content.length;
} catch (VFSException e) {
return 0;
}
}
----
I must admit, that this analysis is a bit sloppy, as the code that I could find did not match the stack traces that I generated with some thread dumps, so my mileage may vary.
Is there any way we can tweak our config so that ReportServer does attempt to read everything into memory?
Offline
#2 2019-06-05 09:10:35
- eduardo
- Administrator
- Registered: 2016-11-01
- Website
Re: sFTP implementation scaling issues
but we do see another thing happening: OutOfMemoryErrors for simple things like doing a file listing.
Some context:
* The reports that we are serving are quite big (read: few hundred Mbs max, with a few of those in the same virtual directory).
* ReportServer CE 3.0.7
* Heap size: 16 Gb
Hi int-mny,
does this happen when you are doing a file listing *when this listing includes the large reports* or does this also happen when listing normal small files / reports ?
And in what form are these large reports in the virtual directory system ? Compiled as PDF or how exactly are they there? How large are these ?
Is there any way we can tweak our config so that ReportServer does attempt to read everything into memory?
I think you mean how to tweak the config so that ReportServer does *not* attempt to read everything into memory, right? 
Regards,
Eduardo
Offline
#3 2019-06-05 13:24:32
- int-mny
- Member
- Registered: 2019-05-28
Re: sFTP implementation scaling issues
We see that when we removes (enought) files from the virtual directory at some point it will return a result but it does take a lot of times. The reports are all CSV files, they differ in size but can be up to a few 100MB. But this behavior is also very consistent with what we see in the code.
File: VirtualContentProviderImpl
@Override
public long getSize(VFSLocation vfsLocation) {
VFSLocation baseLocation = vfsLocation.getVirtualBaseLocation();
if(vfsLocation.equals(baseLocation))
return 0;
(2) return doGetSize(vfsLocation);
}
protected long doGetSize(VFSLocation vfsLocation) {
try {
(0) byte[] content = doGetContent(vfsLocation);
return content.length;
} catch (VFSException e) {
return 0;
}
}Is that for each file it will load all the content in memory to determine the file size. With reports of a few 100MB this will be inefficient and will lead to out of memory issues.
Offline
#4 2019-06-05 13:31:17
- eduardo
- Administrator
- Registered: 2016-11-01
- Website
Re: sFTP implementation scaling issues
Hi int-mny,
thanks for your answer. Please also answer this:
"does this happen when you are doing a file listing *when this listing includes the large reports* or does this also happen when listing normal small files / reports ? "
Regards,
Eduardo
Offline
#5 2019-06-06 09:20:05
- int-mny
- Member
- Registered: 2019-05-28
Re: sFTP implementation scaling issues
I tested 2 directories, one with small files (around 1 second) one with big files took around 1 minute. Even more/bigger reports will timeout/crash.
During a big listing we also see the CPU usage go trough the roof.
And combined the reports are more then a GB. So loading them all in memory to get the file size also does explain the behavior
> time lftp -p 8022 -u root -e "ls /tsreport/xxxxxxxxx/Test_smallfiles;bye" sftp://localhost
Password:
-r--r--r-- 1 owner owner 52 Jun 6 10:48 Test_smallfiles_201906061048
-r--r--r-- 1 owner owner 52 Jun 6 10:56 Test_smallfiles_201906061056
-r--r--r-- 1 owner owner 52 Jun 6 10:57 Test_smallfiles_201906061057
-r--r--r-- 1 owner owner 52 Jun 6 10:58 Test_smallfiles_201906061058
-r--r--r-- 1 owner owner 52 Jun 6 10:59 Test_smallfiles_201906061059
-r--r--r-- 1 owner owner 52 Jun 6 11:00 Test_smallfiles_201906061100
-r--r--r-- 1 owner owner 52 Jun 6 11:01 Test_smallfiles_201906061101
-r--r--r-- 1 owner owner 52 Jun 6 11:02 Test_smallfiles_201906061102
-r--r--r-- 1 owner owner 52 Jun 6 11:03 Test_smallfiles_201906061103
-r--r--r-- 1 owner owner 52 Jun 6 11:04 Test_smallfiles_201906061104
-r--r--r-- 1 owner owner 52 Jun 6 11:05 Test_smallfiles_201906061105
-r--r--r-- 1 owner owner 52 Jun 6 11:06 Test_smallfiles_201906061106
-r--r--r-- 1 owner owner 52 Jun 6 11:07 Test_smallfiles_201906061107
-r--r--r-- 1 owner owner 52 Jun 6 11:08 Test_smallfiles_201906061108
-r--r--r-- 1 owner owner 52 Jun 6 11:09 Test_smallfiles_201906061109
-r--r--r-- 1 owner owner 52 Jun 6 11:10 Test_smallfiles_201906061110
-r--r--r-- 1 owner owner 52 Jun 6 11:11 Test_smallfiles_201906061111
-r--r--r-- 1 owner owner 52 Jun 6 11:12 Test_smallfiles_201906061112
-r--r--r-- 1 owner owner 52 Jun 6 11:13 Test_smallfiles_201906061113
-r--r--r-- 1 owner owner 52 Jun 6 11:14 Test_smallfiles_201906061114
real 0m1.005s
user 0m0.004s
sys 0m0.005s
xxx@xxxx[ ~ ] (11:14:27 - Thu Jun 06):)
xxx@xxxx [ ~ ] (11:14:27 - Thu Jun 06):)
> time lftp -p 8022 -u root -e "ls /tsreport/xxxxxx/Today;bye" sftp://localhost
Password:
-r--r--r-- 1 owner owner 1259925 Jun 6 02:14 201906060214_RPMLTX_ZLCSVR91
-r--r--r-- 1 owner owner 28766 Jun 6 00:00 201906061200_RPMLTX_ZLCSVR01
-r--r--r-- 1 owner owner 53958 Jun 6 00:00 201906061200_RPMLTX_ZLCSVR89
-r--r--r-- 1 owner owner 60801 Jun 6 00:00 201906061200_RPMLTX_ZLCSVR90
-r--r--r-- 1 owner owner 1571 Jun 6 00:00 201906061200_RPMLTX_ZLCSVR93
-r--r--r-- 1 owner owner 32 Jun 6 00:00 201906061200_RPMLTX_ZLCSVRA5
-r--r--r-- 1 owner owner 199 Jun 6 00:01 201906061201_RPMLTX_ZLCSVQI1
-r--r--r-- 1 owner owner 318 Jun 6 00:01 201906061201_RPMLTX_ZLCSVR02
-r--r--r-- 1 owner owner 79032860 Jun 6 00:02 201906061202_RPMLTX_ZLCSVR79
-r--r--r-- 1 owner owner 204 Jun 6 00:03 201906061203_RPMLTX_ZLCSVR68
-r--r--r-- 1 owner owner 55572525 Jun 6 00:04 201906061204_RPMLTX_ZLCSVR23
-r--r--r-- 1 owner owner 47 Jun 6 00:05 201906061205_RPMLTX_ZLCSVR92
-r--r--r-- 1 owner owner 217 Jun 6 00:07 201906061207_RPMLTX_ZLCSVR24
-r--r--r-- 1 owner owner 722 Jun 6 00:08 201906061208_RPMLTX_ZLCSVR04
-r--r--r-- 1 owner owner 60733473 Jun 6 00:10 201906061210_RPMLTX_ZLCSVR50
-r--r--r-- 1 owner owner 192962794 Jun 6 00:11 201906061211_RPEDM_ZLCSVR55
-r--r--r-- 1 owner owner 671 Jun 6 00:11 201906061211_RPMLTX_ZLCSVR31
-r--r--r-- 1 owner owner 52677726 Jun 6 00:30 201906061230_RPMLTX_ZLCSVR53
real 1m0.124s
user 0m0.014s
sys 0m0.019s
xxx@xxxx [ ~ ] (11:15:50 - Thu Jun 06):)
>Last edited by int-mny (2019-06-06 09:24:29)
Offline
#6 2019-06-06 10:27:53
- eduardo
- Administrator
- Registered: 2016-11-01
- Website
Re: sFTP implementation scaling issues
Hi int-mny,
thanks for your analysis.
I totally agree that it would be better not to read the complete file into memory in order just to find out its size. I raised ticket RS-3480 for this.
If you have any ideas / suggestions / specific code changes in order to fix this, these are of course always welcome. Since you already analyzed the code and pointed into the right direction, you may want to further analyze the problem with a solution 
Regards,
Eduardo
Offline
#7 2019-06-06 12:06:38
- int-mny
- Member
- Registered: 2019-05-28
Re: sFTP implementation scaling issues
In order to see if we can provide a solution we would need access to the git repo currently used to build your binaries. The source code downloaded from you site appears to not completely match with the code used for building. All the stack traces point to different line numbers then what we see in the code. Next to that, the pom files are not provided. If you can help us out with better access to the code we can see in how far we can analyze it.
Offline
#8 2019-06-07 07:48:03
- eduardo
- Administrator
- Registered: 2016-11-01
- Website
Re: sFTP implementation scaling issues
Hi int-mny,
at the moment, we have this tutorial for setting up reportserver in eclipse: https://reportserver.net/en/tutorials/dev-30/
But this tutorial has to be actualized for 3.0.7. We have a ticket for this and we will update when we have some time for it.
However, since the SFTP functionality has not changed since 3.0.1, you can use the 3.0.1 code and this tutorial for installing reportserver in eclipse.
Regards,
Eduardo
Offline
Pages: 1