#1 2015-06-19 15:30:04

SteffenF
Member
Registered: 2015-06-19

CSV Datasource with URL Connector http://username:password@...

Hi,

I'm trying to setup a CSV datasource which uses a URL Connector.
Purpose is to get live monitoring status information from a Nagios/Check_MK server which does provide the info as csv.

The access to this URL has to be protected with username and password.

A datasource of type "Relational Database " provides two fields for username and password including the security mechanism to not send back the password to browser.
A datasource of type "CSV List" with "URL connector" does not provide such username and password fields.

I've also tried to include the username:password into URL as "http://username:password@IPofNagiosServer/check_mk/view.py/....", but always receive a 401 Authentication required.

Error message:
"
Error
The report could not be executed: java.lang.IllegalStateException: java.lang.IllegalStateException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://username:password@IPofNagiosServer/check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export
"

On Nagios/Check_MK apache the username+password seem to not arrive. Apache httpd access.log:

172.0.0.2 - - [19/Jun/2015:17:06:09 +0200] "GET /check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export HTTP/1.1" 401 401 "-" "Java/1.7.0_79"


My first assumption:
Username + password in URL are not supported by java.net.URL / java.net.URLConnection and would need to be provided via HTTP header / connection property "Authentication: Basic <encoded username:password>"


I do not see this as a bug, but as an potential improvement/feature to support passwords for URL Connectors and to protect them in a similar way as you do already for Database connections.


Full exception in reportserver tomcat catalina.log:

Jun 19, 2015 5:06:09 PM net.datenwerke.rs.core.service.reportmanager.ReportExecutorServiceImpl execute
WARNING: java.lang.IllegalStateException: java.lang.IllegalStateException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://username:password@IPofNagiosServer/check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export
java.lang.RuntimeException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://username:password@IPofNagiosServer/check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export
        at net.datenwerke.rs.base.service.datasources.helpers.CsvToInMemoryDb.getConnection(CsvToInMemoryDb.java:143)
        at net.datenwerke.rs.base.service.datasources.table.transformers.CsvDatasourceTableTransformer.transform(CsvDatasourceTableTransformer.java:51)
        at net.datenwerke.rs.base.service.datasources.table.transformers.CsvDatasourceTableTransformer.transform(CsvDatasourceTableTransformer.java:1)
        at net.datenwerke.rs.core.service.datasourcemanager.datasourcetransformers.DatasourceDefinitionTransformerManager.transform(DatasourceDefinitionTransformerManager.java:49)
        at net.datenwerke.rs.base.service.reportengines.table.TableReportEngine.doExecuteNormal(TableReportEngine.java:168)
        at net.datenwerke.rs.base.service.reportengines.table.TableReportEngine.doExecuteNormal(TableReportEngine.java:148)
        at net.datenwerke.rs.base.service.reportengines.table.TableReportEngine.doExecute(TableReportEngine.java:90)
        at net.datenwerke.rs.core.service.reportmanager.engine.ReportEngine.execute(ReportEngine.java:127)
        at net.datenwerke.rs.core.service.reportmanager.ReportExecutorServiceImpl.execute(ReportExecutorServiceImpl.java:186)
        at net.datenwerke.rs.core.service.reportmanager.ReportExecutorServiceImpl.execute(ReportExecutorServiceImpl.java:119)
        at net.datenwerke.rs.core.service.reportmanager.ReportExecutorServiceImpl.execute(ReportExecutorServiceImpl.java:106)
        at net.datenwerke.rs.core.service.reportmanager.ReportExecutorServiceImpl.execute(ReportExecutorServiceImpl.java:101)
        at net.datenwerke.rs.base.service.reportengines.table.TableReportUtilsImpl.getReturnedColumns(TableReportUtilsImpl.java:180)
        at net.datenwerke.rs.base.service.reportengines.table.TableReportUtilsImpl.getReturnedPlainColumns(TableReportUtilsImpl.java:144)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl.getReturnedColumns(TableReportUtilityServiceImpl.java:212)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473.CGLIB$getReturnedColumns$1(<generated>)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473$$FastClassByGuice$$997eb26c.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at net.datenwerke.security.service.security.aop.SecurityCheckInterceptor.invoke(SecurityCheckInterceptor.java:110)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at net.datenwerke.gf.service.gwtstacktrace.CatchStacktraceInterceptor.invoke(CatchStacktraceInterceptor.java:38)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473.getReturnedColumns(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473.CGLIB$processCall$11(<generated>)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473$$FastClassByGuice$$997eb26c.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at net.datenwerke.security.service.security.aop.SecurityCheckInterceptor.invoke(SecurityCheckInterceptor.java:110)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at net.datenwerke.gf.service.gwtstacktrace.CatchStacktraceInterceptor.invoke(CatchStacktraceInterceptor.java:38)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473.processCall(<generated>)
        at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
        at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473.CGLIB$service$23(<generated>)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473$$FastClassByGuice$$997eb26c.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at net.datenwerke.security.service.security.aop.SecurityCheckInterceptor.invoke(SecurityCheckInterceptor.java:110)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at net.datenwerke.gf.service.gwtstacktrace.CatchStacktraceInterceptor.invoke(CatchStacktraceInterceptor.java:38)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at net.datenwerke.rs.base.server.table.TableReportUtilityServiceImpl$$EnhancerByGuice$$fd148473.service(<generated>)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
        at com.google.inject.persist.PersistFilter.doFilter(PersistFilter.java:89)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://username:password@IPofNagiosServer/check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export
        at net.datenwerke.rs.core.service.internaldb.InternalDbServiceImpl.getConnection(InternalDbServiceImpl.java:345)
        at net.datenwerke.rs.base.service.datasources.helpers.CsvToInMemoryDb.getConnection(CsvToInMemoryDb.java:132)
        ... 82 more
Caused by: java.lang.IllegalStateException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://username:password@IPofNagiosServer/check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export
        at net.datenwerke.rs.base.service.datasources.helpers.CsvToInMemoryDb$1.open(CsvToInMemoryDb.java:86)
        at net.datenwerke.rs.core.service.internaldb.InternalDbTableCreator.openStream(InternalDbTableCreator.java:79)
        at net.datenwerke.rs.core.service.internaldb.InternalDbServiceImpl.getConnection(InternalDbServiceImpl.java:326)
        ... 83 more
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: http://username:password@IPofNagiosServer/check_mk/view.py?view_name=servicegroup&servicegroup=PROD&output_format=csv_export
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)
        at java.net.URL.openStream(URL.java:1037)
        at net.datenwerke.rs.base.service.datasources.connectors.UrlDatasourceConnector.getDataStream(UrlDatasourceConnector.java:38)
        at net.datenwerke.rs.base.service.datasources.definitions.FormatBasedDatasourceDefinition.getDataStream(FormatBasedDatasourceDefinition.java:46)
        at net.datenwerke.rs.base.service.datasources.helpers.CsvToInMemoryDb$1.open(CsvToInMemoryDb.java:79)
        ... 85 more


Regards

Offline

#2 2015-06-19 15:49:58

SteffenF
Member
Registered: 2015-06-19

Re: CSV Datasource with URL Connector http://username:password@...

Remark:
my current workaround is to load the csv via curl triggered by cron into local filesystem and use for the datasource URL file:///pathtocsvfile

Last edited by SteffenF (2015-06-19 15:50:22)

Offline

#3 2015-06-20 07:38:38

Arno Mittelbach
datenwerke
Registered: 2012-02-14

Re: CSV Datasource with URL Connector http://username:password@...

Hi,

you are right. For HTTP Authentication one indeed needs to do some extra work on the java side which is why this is currently not working. I've added a feature request ticket for this. In the mean time, if you are not happy with the curl based solution you could also do some hack in RS: you could point the URL to a ReportServer script (scriptAccess) and have the script go and get the CSV.

Cheers
-Arno

Offline

#4 2016-11-08 20:52:11

bpeikes
Member
Registered: 2016-10-29

Re: CSV Datasource with URL Connector http://username:password@...

Couple questions about CSV datasources via URL:
1) Has support for http://username:password@aaa.bbb.ccc.ddd/URL been added?
2) Lets say we get a CSV datasource via http set up so that it does not need a username and password, how do you go about testing it?

Offline

#5 2016-11-08 21:09:31

bpeikes
Member
Registered: 2016-10-29

Re: CSV Datasource with URL Connector http://username:password@...

To answer my question above, it does appear that http://username:password@aaa.bbb.ccc.ddd/URL works.

Offline

#6 2020-07-06 11:22:00

itatwork
Member
Registered: 2019-12-11

Re: CSV Datasource with URL Connector http://username:password@...

Hi,

what is the optimal syntax for putting files via curl to the local RS fileserver?

What is the correct Url of a fileserver folder?

Thanks,

Christoph

Last edited by itatwork (2020-07-06 11:33:39)

Offline

#7 2020-07-17 12:33:31

eduardo
Administrator
Registered: 2016-11-01
Website

Re: CSV Datasource with URL Connector http://username:password@...

Hi Christoph,

did you check the SFTP feature? https://reportserver.net/en/guides/admi … TP-Server/
If you install sftp, you can put files easily via SFTP.

Regards,
Eduardo

Offline

#8 2020-08-17 14:14:15

itatwork
Member
Registered: 2019-12-11

Re: CSV Datasource with URL Connector http://username:password@...

Hi Eduardo,

thank you, i will try this.


Cheers,
Christoph

Offline

Board footer

Powered by FluxBB