Performance Issue for Permission Management

ReportServer (EN)
1

Hi,
I have integrated and imported hundreds of users from active directory using below procedure

http://blog.datenwerke.net/2013/08/ReportServer-LDAP-ActiveDirectory-authentication.html

Everything works fine, the problem I am facing during permission management when we have to select “Folk” a pop up window appears for the selection of available users, it takes lot of time to show the available options (users), I believe even if we close the window the action (fetching users info) is running in background. It is really time consuming when we have to assign different permissions to different users.

2

Hi azeemdin,

is it only slow the first time the user selection window opens or is it the same everytime. Could you also give me an estimate of how many nodes you have in the user tree so that we can
try to run some tests.

For the mean time, there is at least a work around to speed things up a little. If you open the permission management popup where you need to select the folk, you can do so by drag and drop.
That is, instead of clicking into the field (which opens the window) go back to the user manager in the administration module and then drag the user or group that you want to select
via drag and drop onto the field.

Cheers
Arno

3

Thanks Arno for prompt reply and work around.

Currently there are 1000 users and two Organization Unit

(I wonder if there is a limit in the script or application to import users from active directory as I could only import 999 users successfully, I still need to investigate, just sharing with you my expereicne, if you can advise somethign in this regard)

4

I have found the issue that limit of number of users is from active directory. I have updated the script to use InitialLdapContext with PagedResultsControl and successfully imported around 12K users. Below is the updated script I have used:

package ldap

import java.util.Map.Entry
import java.util.logging.Level
import java.util.logging.Logger

import javax.naming.*;
import javax.naming.directory.*;
import javax.naming.ldap.*;

import net.datenwerke.security.service.usermanager.UserManagerService
import net.datenwerke.security.service.usermanager.entities.AbstractUserManagerNode
import net.datenwerke.security.service.usermanager.entities.Group
import net.datenwerke.security.service.usermanager.entities.OrganisationalUnit
import net.datenwerke.security.service.usermanager.entities.User




UserManagerService userManagerService = GLOBALS.getRsService(UserManagerService.class);

LdapUserLoader lul = new LdapUserLoader();

lul.setProviderUrl("ldap://directory.example.com:389");
lul.setSecurityPrincipal("CN=ldaptest,CN=Users,DC=directory,DC=example,DC=com");
lul.setSecurityCredentials("ldaptest");

lul.setLdapBase("OU=EXAMPLE,DC=directory,DC=example,DC=com");
OrganisationalUnit targetNode = (GLOBALS.findObject("/usermanager/external"));

if(null == targetNode){
	AbstractUserManagerNode umRoot = userManagerService.getRoots().get(0);
	targetNode = new OrganisationalUnit("external");
	umRoot.addChild(targetNode);
	userManagerService.persist(targetNode);
}

lul.setTargetNode(targetNode);
lul.run();


public class LdapUserLoader {

	private final Logger logger = Logger.getLogger(getClass().getName());


	private String ldapBase;
	private String ldapFilter = "(|(objectClass=organizationalUnit)(objectClass=user)(objectClass=group))";

	private String providerUrl;
	private String securityCredentials;

	private String securityPrincipal;
	private OrganisationalUnit targetNode;

	private boolean includeNamespace = false;

	private Map<String, AbstractUserManagerNode> guidMap;
	private Map<LdapName, AbstractUserManagerNode> nodesInDirectoryByName;
	private Map<String, AbstractUserManagerNode> nodesInDirectoryByGuid;
	private TreeMap<LdapName, SearchResult> searchResults;

	private List<AbstractUserManagerNode> removedNodes;
	private List<AbstractUserManagerNode> addedNodes;



	private class AdGUID {
		byte[] bytes;

		public AdGUID(byte[] bytes) {
			this.bytes = bytes;
		}

		private void addByte(StringBuffer sb, int k) {
			if(k<=0xF)
				sb.append("0");
			sb.append(Integer.toHexString(k));
		}

		@Override
		public String toString() {
			StringBuffer sb = new StringBuffer();
			addByte(sb, (int)bytes[3] & 0xFF);
			addByte(sb, (int)bytes[2] & 0xFF);
			addByte(sb, (int)bytes[1] & 0xFF);
			addByte(sb, (int)bytes[0] & 0xFF);
			sb.append("-");
			addByte(sb, (int)bytes[5] & 0xFF);
			addByte(sb, (int)bytes[4] & 0xFF);
			sb.append("-");
			addByte(sb, (int)bytes[7] & 0xFF);
			addByte(sb, (int)bytes[6] & 0xFF);
			sb.append("-");
			addByte(sb, (int)bytes[8] & 0xFF);
			addByte(sb, (int)bytes[9] & 0xFF);
			sb.append("-");
			addByte(sb, (int)bytes[10] & 0xFF);
			addByte(sb, (int)bytes[11] & 0xFF);
			addByte(sb, (int)bytes[12] & 0xFF);
			addByte(sb, (int)bytes[13] & 0xFF);
			addByte(sb, (int)bytes[14] & 0xFF);
			addByte(sb, (int)bytes[15] & 0xFF);

			return sb.toString();
		}

	}

	public LdapUserLoader() {

	}


	public String getLdapBase() {
		return ldapBase;
	}

	public String getProviderUrl() {
		return providerUrl;
	}

	public String getSecurityCredentials() {
		return securityCredentials;
	}

	public String getSecurityPrincipal() {
		return securityPrincipal;
	}

	public void setLdapBase(String ldapBase) {
		this.ldapBase = ldapBase;
	}

	public void setProviderUrl(String providerUrl) {
		this.providerUrl = providerUrl;
	}

	public void setSecurityCredentials(String securityCredentials) {
		this.securityCredentials = securityCredentials;
	}

	public void setSecurityPrincipal(String securityPrincipal) {
		this.securityPrincipal = securityPrincipal;
	}

	public void setTargetNode(OrganisationalUnit targetNode) {
		this.targetNode = targetNode;
	}

	public boolean isIncludeNamespace() {
		return includeNamespace;
	}

	public void setIncludeNamespace(boolean includeNamespace) {
		this.includeNamespace = includeNamespace;
	}

	public String getLdapFilter() {
		return ldapFilter;
	}

	public void setLdapFilter(String ldapFilter) {
		this.ldapFilter = ldapFilter;
	}



	private Properties compileProperties(){
		Properties props = new Properties();

		props.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
		props.setProperty(Context.PROVIDER_URL, providerUrl);
		//props.setProperty(Context.URL_PKG_PREFIXES, "com.sun.jndi.url");
		//props.setProperty(Context.REFERRAL, "throw");
		props.setProperty(Context.SECURITY_AUTHENTICATION, "simple");

		props.setProperty(Context.SECURITY_PRINCIPAL, securityPrincipal);
		props.setProperty(Context.SECURITY_CREDENTIALS, securityCredentials);

		/* return these as binary */
		props.put("java.naming.ldap.attributes.binary","objectGUID");
		return props;
	}

	private void createGuidMap(AbstractUserManagerNode current) {
		Map<String, AbstractUserManagerNode> map = new HashMap<>();
		createGuidMap(current, map );

		guidMap = map;
	}

	private void createGuidMap(AbstractUserManagerNode current, Map<String, AbstractUserManagerNode> map){
		map.put(current.getGuid(), current);

		for(AbstractUserManagerNode cn : current.getChildren()){
			createGuidMap(cn, map);
		}
	}

	private String getStringAttribute(SearchResult sr, String attributeName) throws NamingException{
		try{
			return sr.getAttributes().get(attributeName).get().toString();
		}catch(Exception e){
			logger.log(Level.WARNING,"failed to retrieve attribute '" + attributeName + "' from " + sr.getNameInNamespace(), e);
			return null;
		}
	}

	private String getGuid(SearchResult sr) throws NamingException{
		try{
			AdGUID guid = new AdGUID((byte[]) sr.getAttributes().get("objectGUID").get());
			return guid.toString();
		}catch(Exception e){
			throw new RuntimeException("failed to retrieve objectGUID from " + sr.getNameInNamespace(), e);
		}
	}

	private void loadFromDirectory() throws NamingException {
		Properties props = compileProperties();
		String originBase = this.providerUrl.endsWith("/")?providerUrl:providerUrl + "/";

		this.nodesInDirectoryByName = new HashMap<>();
		this.nodesInDirectoryByGuid = new HashMap<>();
		this.addedNodes = new ArrayList<>();
		this.removedNodes = new ArrayList<>();

		InitialLdapContext  ctx = null ;

		try {
			ctx = new InitialLdapContext (props);
			
			int pageSize = 500; 
			byte[] cookie = null;
			ctx.setRequestControls([new PagedResultsControl(pageSize, Control.NONCRITICAL) ] as Control[]);

			SearchControls searchControls = new SearchControls();
			searchControls.setSearchScope(SearchControls.SUBTREE_SCOPE);

			LdapName ldapBaseName = new LdapName(getLdapBase());
			

			/* order search results by name to make sure children never get processed before their parent */
			searchResults = new TreeMap<>();
			int total;
			
			while(true) {
			NamingEnumeration<SearchResult> results = ctx.search(ldapBaseName, this.ldapFilter, searchControls);
				while (results.hasMore()) {
					SearchResult sr = (SearchResult) results.next();
					searchResults.put(new LdapName(sr.getNameInNamespace()), sr);
				}
				
				Control[] controls = ctx.getResponseControls();
				if (controls != null) {
					for (int i = 0; i < controls.length; i++) {
						if (controls[i] instanceof PagedResultsResponseControl) {
							PagedResultsResponseControl prrc = (PagedResultsResponseControl)controls[i];
							//total = prrc.getResultSize();
							//logger.log(Level.INFO,"Total is: " + total.toString());							
							cookie = prrc.getCookie();							
						}
					}
				} else {
					logger.log(Level.INFO,"No controls were sent from the server");
					break;
				}
				
				ctx.setRequestControls([new PagedResultsControl(pageSize, cookie, Control.CRITICAL) ] as Control[]);
				
				if(null == cookie)
				break;
			}
			
			for(SearchResult sr : searchResults.values()){
				try {
					LdapName nodeName = new LdapName(isIncludeNamespace() ? sr.getNameInNamespace() : sr.getName());
					LdapName nodeNameInNamespace = new LdapName(sr.getNameInNamespace());

					/* skip empty nodes */
					if(nodeName.size() == 0)
						continue;


					/* get parent node */
					LdapName parentName = (LdapName) nodeNameInNamespace.getPrefix(Math.max(0, nodeNameInNamespace.size() - 1));
					AbstractUserManagerNode parent = this.nodesInDirectoryByName.get(parentName);
					if(null == parent){
						parent = targetNode;
					}

					/* create node */
					Attribute objectClass = sr.getAttributes().get("objectClass");
					AbstractUserManagerNode umNode = null;
					if(null != objectClass) {
						if(objectClass.contains("organizationalUnit")) {
							umNode = createOUNode(sr, parent);

						} else if(objectClass.contains("user")) {
							umNode = createUserNode(sr, parent);

						} else if(objectClass.contains("group")){
							umNode = createGroupNode(sr, parent);
						}	
					

					if(null != umNode) {
							/* set common attributes */
							umNode.setWriteProtection(false);
							umNode.setGuid(getGuid(sr));
							umNode.setOrigin(originBase + sr.getNameInNamespace());

							nodesInDirectoryByName.put(new LdapName(sr.getNameInNamespace()), umNode);
							nodesInDirectoryByGuid.put(getGuid(sr), umNode);
						}
					}
				}catch(Exception e){
					throw new RuntimeException("Error processing search result: " + sr.getNameInNamespace() , e);
				}
			}


		}finally{
			try {
				if(null != ctx)
					ctx.close();
			} catch (NamingException e) {
				logger.log(Level.WARNING, e.getMessage(), e);
			}
		}
	}


	private AbstractUserManagerNode createGroupNode(SearchResult sr, AbstractUserManagerNode parent) throws NamingException {
		Group node = (Group) guidMap.get(getGuid(sr));
		if(null == node){
			node = new Group();
			addedNodes.add(node);
		}
		parent.addChild(node);

		/* copy Group attributes */
		node.setName(getStringAttribute(sr, "name"));

		return node;
	}


	private AbstractUserManagerNode createUserNode(SearchResult sr, AbstractUserManagerNode parent) throws NamingException {
		User node = (User) guidMap.get(getGuid(sr));
		if(null == node){
			node = new User();
			addedNodes.add(node);
		}
		parent.addChild(node);

		/* copy User attributes */
		node.setFirstname(getStringAttribute(sr, "givenName"));
		node.setLastname(getStringAttribute(sr, "sn"));
		node.setUsername(getStringAttribute(sr, "sAMAccountName"));

		return node;
	}

	private AbstractUserManagerNode createOUNode(SearchResult sr, AbstractUserManagerNode parent) throws NamingException {
		OrganisationalUnit node = (OrganisationalUnit) guidMap.get(getGuid(sr));
		if(null == node){
			node = new OrganisationalUnit();
			addedNodes.add(node);
		}
		parent.addChild(node);

		/* copy OU attributes */
		node.setName(getStringAttribute(sr, "name"));

		return node;
	}


	private void postprocessGroups() throws NamingException {

		/* clear */
		for(Entry<LdapName, AbstractUserManagerNode> entry : nodesInDirectoryByName.entrySet()){
			if(entry.getValue() instanceof Group){
				Group group = (Group) entry.getValue();
				group.getUsers().clear();
				group.getOus().clear();
				group.getReferencedGroups().clear();
			}
		}

		/* add appropriate users */
		for(Entry<LdapName, AbstractUserManagerNode> entry : nodesInDirectoryByName.entrySet()){
			if(entry.getValue() instanceof Group){
				Group group = (Group) entry.getValue();
				SearchResult sr = searchResults.get(entry.getKey());
				if(null != sr.getAttributes().get("member")){
					NamingEnumeration<?> members = sr.getAttributes().get("member").getAll();
					while(members.hasMore()){
						LdapName memberName = new LdapName(members.next().toString());
						AbstractUserManagerNode member = nodesInDirectoryByName.get(memberName);
						if(null != member){
							if(member instanceof User)
								group.addUser((User) member);
							if(member instanceof OrganisationalUnit)
								group.addOu((OrganisationalUnit) member);
							if(member instanceof Group)
								group.addReferencedGroup((Group) member);
						}
					}
				}
			}
		}
	}

	private void printTree(AbstractUserManagerNode current){
		StringBuilder sb = new StringBuilder();
		List<AbstractUserManagerNode> rl = current.getRootLine();
		Collections.reverse(rl);
		for(AbstractUserManagerNode node : rl){
			sb.append(node.getName()).append(".");
		}
		sb.append(current.getName() + " [" + current.getClass().getSimpleName() + "]" );

		if(current instanceof Group){
			Group group = (Group) current;
			sb.append(" (").append(group.getUsers().size() + group.getOus().size() + group.getReferencedGroups().size()).append(" members)");
		}

		System.out.println(sb.toString());

		for(AbstractUserManagerNode cn : current.getChildren()){
			printTree(cn);
		}
	}


	private void deleteRemovedUsers(AbstractUserManagerNode current) {
		for(AbstractUserManagerNode c : current.getChildren()){
			deleteRemovedUsers(c);
		}

		if(null != current.getOrigin() && current.getOrigin().startsWith(providerUrl) && !nodesInDirectoryByGuid.containsKey(current.getGuid())){
			current.getParent().removeChild(current);
			removedNodes.add(current);
		}
	}

	public void run() throws NamingException{
		createGuidMap(targetNode);

		loadFromDirectory();
		postprocessGroups();

		deleteRemovedUsers(targetNode);

		//logger.log(Level.INFO,"Retrieved nodes from directory: " + nodesInDirectoryByGuid.size() );
		//logger.log(Level.INFO,"Nodes added: " + addedNodes.size() );
		//logger.log(Level.INFO,"Nodes removed: " + removedNodes.size() );
		int overallCount = countNodes(targetNode) - 1;
		//logger.log(Level.INFO,"Overall nodes in rs: " + overallCount);

		if(overallCount != nodesInDirectoryByGuid.size())
			throw new RuntimeException("Failed to import user data from directory");
		else
			logger.log(Level.INFO,"done.");

		//		printTree(targetNode);
	}


	private int countNodes(AbstractUserManagerNode current) {
		int i = 1;
		for(AbstractUserManagerNode n : current.getChildren()){
			i += countNodes(n);
		}
		return i;
	}

}

The only problem is now the performance with huge number of users, it is difficult to search and edit as it takes a lot of time to load specific user info in “User management”

5

Hi azeemdin,

thank you for looking into this. If thats fine with you, I’ll update the blog post to include your modifications.

We never actually tested rs with that many users and did not put much effort into the optimization of the user manager. There are some rather small modifications that should make a significant improvement. We will look into this and let you know.

Kind regards,
Thorsten

6

Hi azeemdin,

could you try the following. Go to the fileserver and rename file /etc/security/passwordpolicy.cf.disabled to passwordpolicy.cf.
Then open the Terminal (CTRL+ALT+T) and execute command “config reload”.

This will activate the password policy (note that this has some effects on how passwords need to be chosen
and that, for example, users will be blocked if they enter the password wrong too often). You can find more information
in the config guide.
However, activating the password policy should yield some performance improvements for the popup loading (sounds strange, but its true).
Happy to give details, in case you are interested.

The improvements are not great but it should make it better. We have made further changes which we ship with the next release.

Cheers
Arno

7

Hi Thorsten /Arno,
Thanks for your support, I tried the configuration you proposed but still it is very slow (I think 12K is unexpected figure as you mentioned).
The other workaround which Arno purposed earlier is much better but for both scenarios I believe search option is better instead of loading all users in one shot for tree view (it also impacts the browser performance).

8

Hi azeemdin,

with the next version the user tree is loaded only once (unless you do a manual reload) and for 10k users this took about 10 to 15 seconds. But I am happy
if the workaround works for you.

Cheers
Arno

9

Hi Arno,
Thanks for support.

Script execution is really fast while importing user but when I tried to remove these users from User Management, it takes again long time so I tried again with script and it really helped and removed complete Organisational Unit and all child nodes but I am not sure if this is the right way of doing it, below is the script I have used

import net.datenwerke.security.service.usermanager.entities.AbstractUserManagerNode
import net.datenwerke.security.service.usermanager.entities.OrganisationalUnit


private void deleteRemovedUsers(AbstractUserManagerNode current) {
	for(AbstractUserManagerNode c : current.getChildren()){
		deleteRemovedUsers(c);
	}

	if(null != current.getOrigin()){
		current.getParent().removeChild(current);
	}
}

OrganisationalUnit targetNode = (GLOBALS.findObject("/usermanager/external"));

if (null != targetNode) {
	deleteRemovedUsers(targetNode);
}
10

Hi azeemdin,

we are currently preparing the ReportServer script guide, so I guess you have something to look forward to. As for supporting custom scripting it is our policy to offer this service only to our support customers (see ReportServer Support Package).

Anyhow. You should be careful with the script that you proposed as this would only remove the OU from its parent, but not actually delete it from the database which might cause a corrupted database. What you would use in this situation is the UserManagerService, a service class that allows you to perform various operations on users, groups, etc. The script could, for example, look like

import net.datenwerke.security.service.usermanager.UserManagerService

def userService = GLOBALS.getRsService(UserManagerService.class)

def targetNode = (GLOBALS.findObject("/usermanager/testOU"));

if (null != targetNode) {
    userService.remove(targetNode);
}

The other option that you have, which is probably the easier one, is to simply use the terminal command rm. That is, you could also simply run the following command on the terminal

rm /usermanager/testOU

In case testOU has child objects it would be

rm -r /usermanager/testOU

I am also a bit surprised that the performance within the user manager is not good for you. Our tests with 10k users were quite satisfactory. However, we split up the 10k users over 100 folders (with 100 users each). Do you, by any chance, import all the 12k users into a single folder?

Cheers
Arno

11

hi Arno,
I really appriciate your time and support, thanks for the details regarding user management. Yes I have imported around 7K users under one folder and 5K users in second, may be splitting will make the difference, I also tried 1000 users in one folder and performance with that was better but not as expected. Currently I have removed all the users and only importing/creating selected one to explore other features of the product.