Collecting data from an Office 365 source

    Indexing an Office 365 source

    This document should be read along with the information listed in Section 10.2 of the Intella User Manual in regards to connecting to an Office 365 source.

     

    Connect to Online Office 365

    The Office 365 source type allows for retrieving both user account and user groups. For each user account used to access Office 365, the source can retrieve data from Outlook, OneDrive, and SharePoint. For each user group, the source retrieves titled conversations containing emails.

    For Outlook, the source retrieves all folders (both standard and user-defined) and all emails therein. For OneDrive, the source retrieves all folders and all files. 

    The Office 365 source uses the Microsoft Graph API to connect to Office 365 and retrieve its resources. There are two modes in which a connection for Office 365 can be made: as a user, and as an application. A user login allows for the retrieval of that user’s resources only. An application login allows for the retrieval of the resources of selected or all users. 

    Depending on the connection type used, the Microsoft Graph service uses a different group of permissions and settings to control access to the Office 365 resources. However, for both connection types, it is required to grant admin consent after assigning the permissions.

    Before using Intella to index the Office 365 data, you need to configure settings in the Azure portal so that Intella can access the Office 365 source. Intella can only access the resource if there are adequate permissions to access the account and content. Below are some issues that have been reported to us. These issues are all related to the Azure portal where the setting and/or permissions have not being set correctly.

    • You get an error (similar to below) when using the 'Connect to Office 365' option in Intella's Add new source wizard.

    • The 'Connect to Office 365' option in the Add new source wizard completes successfully, but you can not index any data from the account.
    • You can connect to the account with no issues, and you can index data in the account with no issues, but Intella indexes other data associated with the account which should not be indexed.
    • You are not able to authenticate due to “access token” or credential issues.  

    Note: Intella will be able to index an Office 365 source provided that the permissions are set correctly. The Office 365 permissions are to do with the account on the Azure portal, and therefore, the user must manage these connectors and permission themselves. This is something that we can not do as we are not Azure administrators. You may need to consult your Azure administrator for more advanced configuration and permission settings to allow access to the Office 365 source. 

    Below are two guides. Thises show how to configure Azure to grant Intella access to Office 365 as a user, and as an application.

    Please note that the 'connect as a user' guide below is a conservative measure to ensure that Intella has ample permissions to access all aspects of the account, so that all data can be indexed. This guide may have more permissions/access set in Azure than what is minimally required for the purpose that you want to use. It is up to the user to set the correct permissions for the required access to the Office 365 account within the Azure portal. 

    Note that Intella reads the data through the Microsoft Graph API. No write permissions are required, and Intella does this in read only mode.


    Configure Azure to connect to an Ofice 365 source as a user 

    Step 1:
    Go to https://portal.azure.com and login using the Office 365 admin credentials.


     

    Step 2:
    Select Azure Active Directory option in the sidebar menu. The Active Directory overview page will be shown.

     

    Step 3:
    In the subsequent sidebar submenu select the App registrations option.


     

    Step 4:
    Click the New application registration button.


     

    Step 5:
    The Create form will be shown.
    Enter a name for your application in the Name field (e.g. Office365crawler).
    In the "Redirect URI" section, select Native in the drop-down list.
    Set an artificial redirect URL in the second field (e.g. https://localhost/office365/crawler).

    Finally, click on the Register button.


     

    Step 6:
    The newly created application will appear in the App Registrations result table, as shown in the following figure.
    Click on the application name in order to see the application's Properties page.


     

    Step 7:
    The Properties for the new application will be shown.
    This page shows the Application ID. The Application ID is required by Intella when connecting to a Office 365 source.
    Record this ID as we will need it later when adding the source in Intella.


     

    Step 8:
    Click on the View API Permissions button, as indicated by the arrow in the screen above.
    By default there is a User.Read privilege added for the Microsoft Graph API.
    We now need to add additional access permissions to the APIs for Microsoft Graph and Office 365 Online.
    Click the Add a permission button.


    A new panel will open after pressing the Add a permission button.
    Click on the Microsoft Graph option.


     

    Step 9:
    Select the Delegated permissions option from the list.


     

    Step 10:
    The list of permissions will appear in the "Select permissions" section of the "Request API permissions page".

    For connecting to an Office 365 source, set the following permissions:


    Permission groups:                                          Permions Granted:

    Calendars                                                     Calendars.read

    Contacts                                                       Contacts.Read

    Mail                                                               Mail.Read

    Organization                                                Organization.Read.All

    Sites                                                              Sites.Read.All

    User                                                              User.Read.All



    By design, Intella invokes only Read and Sign in operations, so no data in Office 365 will be changed, even if Write permissions are chosen. However, Read and Read.All privileges must be granted in order to allow Intella to download the corresponding elements.

    Note: These are the correct permission settings at the time of this writing. Over the last year we have seen several changes to the Azure management portal and related permissions. If you have access or connection issues, you may not have granted enough permissions to access the source. You should troubleshoot this by checking whether new permissions, relevant to Office 365, have been added to Azure, or whether there have been any changes to the current permissions mentioned above.

    Finally, click the Update permissions button to complete the configuration of the permissions for the selected API.


     

    Step 11:
    After adding all required permissions to the list, you need to grant admin consent for the permissions.
    Please click on the 'Grant admin consent' button on the 'API permissions' page.


     

    Step 12:

    Next go to the Authentication settings, scroll down to Advanced Settings and set the Allow public client flows to Yes


    Step 13:
    Now that you have configured access to a Office 365 account, you can use the Add new wizard, with the 'Office 365' option, in Intella to index the Office 365 data. Remember that you will need your login credentials, and the Application ID to connect to a Office 365 source.
     

     

     

    Configure Azure to connect to an Ofice 365 source as an application 

    Step 1:
    Go to https://portal.azure.com and login using the Office 365 admin credentials. Then click on the Azure Active Directory service.


    Step 2:

    We need to create a new application for the crawler. First click on App Registrations, then click on New Registration to create the application.

     

    On the Register application page, enter a name for the application, then click on the Register button at the bottom of the page.


     

    Step 3:

    You will see that the new application has been created. First copy the Application ID and the Directory tenant ID to a text file. We will need these IDs later when we use Intella to connect to the source.

    Once that is done, click on the Add a Redirect URI link on the right hand side. 


     

    We need to add two platforms for this connector. Click on the Add platform button. First setup the Web platform by clicking on this option on the right hand side.

     

    For the Web platform, enter a redirect URL in the top field. This can basically be anything. Next, check the AccessTokens box at the bottom, then click on the Configure button.

     

     

    Once done you will automatically go back to the Platform configurations page. Now click on the Add platform button again, this time add the Mobile and Desktop Applications option on the right.

     

     

    You can use the same redirect URL that you used for the Web platform here, but make sure that you make it HTTPS. Once done, click on the Configure button.

     

     

    Scroll to the bottom of the Platform configuration page and set the Treat application as a public client setting to Yes.

     

    Once done, scroll to the top and click the Save button to save the configuration.

     

    Step 4:

    Now we need to create a password. From the menu on the left, click the Certificates and Secrets option. Now click on the New Client Secret option to generate a password.

     

    A dialogue will be shown. Enter a description (which can be anything), then click on the Add button.

     

    Now copy the password to your text file so that we can use it later when connecting with Intella.


    Step 5:

    The next step is to add the permissions required to access and index the Office 365 data. Click on the API Permissions option in the left menu. By default the User.Read permission is enabled. Remove this permission by clicking on the three dots to the right, and clicking on Remove permission.

     

    You will need to confirm that you want to remove the permission.

     

    Once done, we need to add the first permission by clicking on the Add a permission button. From the right hand side, click on the Microsoft Graph option.

     

    Click on the Application permissions option on the right.


    Scroll down to the bottom of the permissions list and open the User folder. Check the User.Read.All permission, then click on the Add permissions button.

     

    We now have our first permission. But, at this point it does not have Admin Consent. Click on the Grant Admin Consent for XXX button to give consent. Again you will need to confirm this change.

     

    Once done, the Status column will show that consent has been granted.

     

    Step 6:

    The User.Read.all permission will allow you to use Intella to connect to the source , and to see some of the folders. Before we go any further with other permissions, we need to first test that what we have done do far is working. If it is working than we will add the other required permissions. If it is not working, then you may have made a mistake in the Azure configuration.

    Open Intella, and click on the Add new source option. Select the Office 365 option and click on Next. This screen is where you enter the credentials that we captured when we were working in Azure. 

    • For the username, first enter the Application ID. Add the colon character (e.g. : ) to the end of this ID. Now enter the Directory tenant ID.
    • Use the password which was saved earlier in the password field.
    • Then enter the Application ID in the Application ID field.

    Click on the Connect to Office 365 button to test the connection.

     

    If the credentials are correct and the settings we configured in Azure are correct, then you should get a 'Successfully connected' message. This means that so far everything has been setup correctly. You wont be able to access any information in Office 365 as we have not setup the other permissions yet. Cancel out of the Add source wizard, and setup the required permissions in Azure before continuing.

    If you do not get this message, then something is not correct with your Azure configuration. That will need to be fixed before going any further.

     

    Step 7:

    Back in the Azure portal, navigate to the crawler application that we created. Click on the API Permissions option from the left hand menu. You will need to add additional components and permissions to gain full access to the Office 365 data. Below is a list of the permission which we have tested and have gained full access to the Office 365 source. Note that most permissions are under the Microsoft graph component. If you are collecting SharePoint data, you will need to add the SharePoint component, and its related permissions. 

     

    The permissions can be added in the same way that we added the User.Read.All permission earlier. Note that you can add more than one permission at a time for one component. Once the permissions have been added, you will need to grant admin consent for them. Once the permissions have been added, go back to Intella and add the Office 365 source to your case.

     

    Other notes:

    • As of this writing, the listed permissions and components allow access to One Drive, Mail, and SharePoint. Note that in the future, Microsoft may make changes to the settings and look of the Azure portal. This means that our screenshots in this document may look different, or there could be more (or less) permission shown in Azure.  
    • If you have followed this guide and your still have issues accessing the Office 365 data, then there may be a few reasons for this.
      1) You may have some special account settings that you may be unaware of. You should first consult with your internal Azure manager. Then contact Microsoft support if you can not resolve the issue. 
      2)  Microsoft may have made some changes where the listed permission no longer work. You should contact Microsoft support if you can not resolve the issue.
    Published