ITpipes and ESRI Documentation

ITpipes and ESRI Documentation

 Table of Contents

​Introduction

ITpipes integration with an ESRI geographic information system (GIS) is a powerful way to leverage the capabilities of these advanced systems. This document covers various components of this integration.  The purpose of this document is to leverage GIS data and ITpipes inspections within the two software platforms.  

Section 1 - Introduction to ITpipes and GIS

This section reviews ITpipes and ESRI GIS integration components, terms, and basic functions.
  1. ITpipes Web - This web-based software (used in Chrome, Firefox, etc.) is used to view, edit, export, and manage underground assets, inspections, observations, and media files. Generally, ITpipes Web is used primarily in the office to review completed inspections.
  2. ITpipes Mobile - This desktop software is installed on the computer in the CCTV truck that will be performing the inspections and fieldwork.  Generally, ITpipes Mobile is used primarily by the field staff to perform inspections in the field.
  3. ITpipes Host & ITpipes Sync - This desktop software is installed on the CCTV truck, and facilitates the transferring of the data between the CCTV truck and the ITpipes database.  Generally, ITpipes Host/Sync are used by the field staff when they come back to the shop at the end of the day to upload all of the inspections, videos, etc. from the day to the main database. 
  4. Web Map - This is an interactive display of geographic information.  This information generally represents the physical location of an asset, like, where the sewer line is, where a lateral is, or where the manhole is.  This information can also represent the attribute information like the size, condition, material, and other information.  In asset management and sewers, a web map might display data like sewer lines, manholes, catch basins, outfalls, laterals, and related data.  The web map could also display the attribute data for the asset.  Generally, this attribute information shows up in a popup when the asset is selected. 
How ITpipes (Web, Sync, Host, and Mobile) and ESRI GIS function

  1. ITpipes Web or Mobile, when configured to use AGOL, is using oAuth services only.

  2. Grouping of layers is not supported when using a Map-Server connection. (Grouping not supported in Feature Services.)

  3. If using ArcGIS Server or Enterprise, ITpipes Software is able to use token authentication via a GIS server user account. 

  4. Esri ArcGIS Software versions that are supported:

    • ArcGIS Enterprise version 10.2 or newer.

    • ArcGIS Server version 10.2.2 or newer.

  5. Only consumes the ESRI feature service and does not update attribute information on the feature service.*

  6. ITpipes Web consumes the feature service and does not update attribute information on the feature service.

  7. ITpipes Web default setting is the ESRI Streets base map. Changing the basemap requires contacting techsupport@itpipes.com.

    1. TPK or VTPK are used as a basemap to show underneath the layers in ITpipes Mobile. To minimize the file size, please use a vector tile package.  TPK or VTPK should not contain any layers, just the basemap.

GIS Data Sources - A client GIS will usually be either an ArcGIS Online account, or an ArcGIS Server.  
  1. Organizations who use ArcGIS Online store their data in the ArcGIS Online cloud, meaning, the data resides within their ArcGIS Online account. 
  2. Organizations who use ArcGIS Server to store the data locally on a server, typically have an internal IT department manage it.  In that case, there is a web portal where s GIS data will be accessed through the web.  This is commonly just called "Portal" for short. 
  3. Another common configuration of GIS is to use ArcGIS Server, and to store GIS data in the cloud with a 3rd party like Amazon, Microsoft, or others.  In this configuration, there is a web portal site, just like if the GIS data in-house on an internal server.
Web Data Layers -- When sewer and related asset data are brought into GIS, this will be in one of two types of "layers".  It's important to understand the difference between these two types of layers, because only an ESRI Feature Service will work with ITpipes.  We recommend reading this ESRI article on Sharing Web Layers before proceeding with this document.
  1. Feature Layer
          
    1. When a Feature Layer is shared, it exposes a Feature Service
    2. This feature service works with ITpipes software     
    3. Example of a feature service:  https://<catalog-url>/<serviceName>/FeatureServer  
    4. This is a type of setup allows GIS data to be available for use over the internet
    5. Can either be "hosted" (stored) within an ArcGIS Online account, or "published" from ArcGIS Server
    6. Data can be edited
    7. Data can be exported, shared, and copied
    8. Click here to read more about ESRI Feature Services 
  2. Map Image Layer 
                  
    1. When a Map Image Layer is shared, it exposes a Map Service
    2. ONLY WORKS with ITpipes WEB/CLOUD software   /
    3. Example of a map service: https://<catalog-url>/<serviceName>/MapServer
    4. This is a type of data that allows GIS data to be available for use over the internet   
    5. Data cannot be edited
    6. Data cannot be exported
    7. Click here to read more about ESRI Map Services
3. Web Map - Within the GIS feature layer, the checkbox for "Enable Sync" will need to be enabled.  This is independent of the program ITpipes Sync that is mentioned earlier in this document. 

Section 2 - Connection Types

The following map types can be used with ITpipes:
  1. AGOL with an Application ID and secret - to use for authentication.
    1. To create an App ID and Secret, here is a short video.
    2. To use AGOL and ITpipes, it is required that AGOL be setup correctly for this connection.
  2. Follow the steps here in this guide for Setting Up ArcGIS Online to Work with ITpipes before proceeding.
    1. Published Feature Services on ArcGIS Server - with no authentication required.
    2. Published Feature Services on ArcGIS Server - with token authentication enabled.
    3. Published Feature services on a federated server with ArcGIS Portal. (Coming in a new ITpipes Web Release Only; this is not available in ITpipes Mobile.)

Section 3 - Layers to Include

When setting up ITpipes and ESRI GIS integration, determining the layers to include is important.  This decision affects what the end-users have displayed inside ITpipes.  For example, for sewer assets, if a user is performing mainline inspections, having manhole detail available to the inspection crew is valuable and influences functionality available with the mainline inspections. 


Minimum GIS Layers

Every client’s system is unique, but at a minimum we recommend having the layers that correspond to the module(s)  purchased. For example, if the ITpipes Mainline Module is purchased, we recommend including sewer mainline GIS data.  Likewise, if the ITpipes Manhole Module is purchased, we recommend including manhole GIS data.   


Optional GIS Layers
Other GIS layers to consider using in this integration are shown, but are not required:

Sewer/Wastewater Assets

  • Gravity Mains

  • Laterals/Taps

  • Lift Stations

Stormwater Assets

  • Conduits

  • Structures

  • Cleanouts 

  • Culverts

Section 4 - Requirements

If using ArcGIS Online, an ESRI ArcGIS Online Organizational account is required.  For all Maps:
  1. All ESRI requirements must be met and fully functional for the maps from ESRI to work in ITpipes; including the Esri software version, and the database connection must be compatible.
  2. Every asset must have a unique ID.  This unique ID must be unique across all of the layers, meaning a pipe segment that has a unique ID of ST-15_ST-16 cannot have a manhole with the exact same unique ID.  
  3. A manhole on either side of this pipe segment could be named ST-15 and ST-16, and that is acceptable as long as no two features (regardless of the layer) have the exact same identical unique ID. 
  4. The map extent should be set to fully cover only the area in which all the assets are available. Instructions from ESRI on how to set up the full extent are available here.
  5. ITpipes suggests using WGS 1984 Web Mercator for the coordinate system of the data.  
  6. Other coordinate systems may work:
  7. ITpipes suggests using WGS 1984 Web Mercator because this is the required coordinate system to utilize the optional Plot function that can be used to plot observations in GIS. It is also required for the Mobile GPS tool to function properly.
  8. Each Pipe asset must have a US & DS Manhole ID.

Feature Service Requirements

  1. Query and Sync options need to be enabled on the feature service when publishing from ArcMap or Pro
  2. To enable the Sync capability on a GIS feature service, after publishing the feature service, watch this short video here to enable the Sync function within the feature service
  3. For integration between ITpipes and GIS (or asset management systems), the data field and attribute data inputs must be set to match the ITpipes template.  
  4. Often, the template used in ITpipes is a compliance standard for pipe inspection that CANNOT be modified per ITpipes contract with the compliance agency.  
  5. In that case, for integration with GIS, the GIS schema must have domain inputs that match those in the ITpipes template.  [See this link for more information on domain values.]

Offline Maps Requirements

  • This is for using GIS maps in the CCTV Truck (using the "ITpipes Mobile" software)

  • Often clients do not have internet access in the field in their CCTV trucks while performing field inspections. ITpipes Mobile software has the capability to download (make a static copy) the GIS infrastructure data for use in the field within the ITpipes Mobile application when no internet is available. This capability makes a geodatabase copy of the data GIS, and sends it to the hard drive on the CCTV vehicle computer, making it available offline wherever the CCTV truck is performing inspections.

    • Requires the "Allow Sync" option in the ESRI feature service to be enabled.

    • In order to download the geodatabase file, a copy of the data has to be created of the feature service. This copy happens each time the ITpipes Sync tool is run. This is why "sync" needs to be enabled within the feature service properties.  Please watch this short video here to learn more.

    • Note: 

      • ITpipes Mobile does not update the Feature Service when using the ITpipes Mobile Sync function. 

      • This allows the ITpipes Mobile software in the truck to get the most up-to-date information from the Feature Service.

ITpipes Plot Observation Feature Requirements

  • Requires the following to function properly. To check out features of that function see this video at time 00:45.

  • Each pipe asset must have a US & DS Manhole ID; if the mainline layer does not have a specific start/end node, the plot feature will not work correctly.

  • The map must use the coordinate system WGS 1984 Web Mercator (3857); if the published map is not Web Mercator then some mapping features may not work correctly. Other measurement systems can be used, but plot observation and possibly other mapping features will not be functional.

    • NOTE: some ESRI documentation may include “(auxiliary sphere)” in the description of this coordinate system.

ITpipes "Create Work Order" Function Requirements

  • This is an internal ITpipes function, and separate from any asset management system like Cartegraph, Cityworks, Lucity, Maximo, Infor.

    • To use an integrated work order function with an asset management system, an additional setup/configuration must be performed.  Contact your ITpipes project manager for more information. 

  • The map must use the coordinate system WGS 1984 Web Mercator (3857); if the published map is not Web Mercator then some mapping features may not work correctly. Other measurement systems can be used

  • Requires the "Allow Sync" option in the ESRI feature service to be enabled.

  • The Unique ID Field must be "Type=String". If the unique ID Field is not string, the create work order function integration to an asset management system will not be functional.


Optional

  1. Flow Arrows: Often field inspectors request the flow arrows to appear on the map viewed inside ITpipes Mobile. This can be done via ESRI products,  instructions are available here.

Section 5 - Integration Methods

There are 2 GIS integration methods with ITpipes Mobile, download and manually attach. Here is a comparison of the two methods: 
  • Download
  • Pro
    • Basemap and geodatabase can be refreshed regularly through ITpipes Mobile
    • Can be maintained through AGOL
    • Uses existing feature service
      • Tiled, Sync, Query enabled
  • Con
    • Level 18 is the maximum layer we can download from (tends to be below 150,000 tiles)
    • Puts zoom limitations in place
    • Not compatible with ArcGIS Server
2.  Manual
    1. Pro
  • Allows for packages with more tiles.
  • Better basemap zoom capabilities.
  • Con
    • Static map
    • Must be generated
    • GIS Department
    • ITpipes can generate a TPK as a service*

Section 6 - Map Configuration in ITpipes Mobile

ITpipes Mobile Map Setup Instructions

Introduction

ITpipes Mobile has the ability to display a downloaded map alongside the inspection data. This allows for additional search options and attribute data synchronization between
the map and ITpipes Mobile. This guide provides instructions for how to enable mapping in an ITpipes Mobile project.

Prerequisites
  1. ITpipes Mobile v10225 or later
  2. An ArcGIS Server or AGOL hosted Feature Server URL, with credentials, provided by your organization
  3. A basemap connection and credentials, either provided by your organization, or by ITpipes 
  4. Initial Connection
Instructions
  1.     In ITpipes Mobile, navigate to the project page. Select a project and the Map Settings section is displayed in the upper right corner. There are different map settings per project


      2.    Once the correct project is selected, check to see if a map is already connected.
                  a.   If a map is already connected, the existing credentials will need removed to add a new map.
                  b.  Select the edit button      and then select the delete button    

      3.    Select the add button     and enter a name for the new map. It can be named anything.

      4.    For the Feature, make sure Download is selected, and then enter the connection information to the map feature server provided by your organization.


      5.    For the Base Map, make sure Download is selected, and then enter the connection information to the basemap provided by ITpipes or your organization.
      6.   Click the connect   button. This will pull in the different Layers of the Feature Server below.

Field Mapping:   Determine which layers will be visible in the map and which attributes will transfer into ITpipes Asset fields.

      7.   For any layer to be visible, select that layer and check the visible box. Then select a corresponding Asset Type, either Mainline, Manhole, or Lateral


      8.    For the ID Field, select the unique identifier that will be different for each object. This field will correspond to the ML_Name or Pipe Segment Reference in PACP, for example

      9.    Expand the Field mapping section and choose matching ITpipes fields for each GIS field. The only required field that must be mapped is the ID field. All other fields are opt


      10.   When field mapping is completely populated, scroll back to the top and select the save button   
      11.   Once the settings are saved, select the refresh button    . This will download the map. After the download, the map will now be visible in the project.

Additional Notes
  1. The map should now be set up in the selected project. The map field mappings can be modified by selecting the edit button in the Map Settings area again.
  2. If a user needs to find the map files or reattach the map locally, the Local Path at the top of the Map Settings section displays the location.
  3. If there any challenges with the map or setup, checking the log files may be valuable. They are located here: C:\Program Files (x86)\Infrastructure Technologies\ITpipes Mobile

Section 7 - Map Configuration in ITpipes Web


These instructions are for adding GIS layers to ITpipes web.

Follow the step-by-step instructions below, or click here to watch a short step-by-step video.  Only the most basic labeling and symbology are needed. Ipipes integration will relabel the assets with the ID field we identify in the field mapping so that CCTV crews and supervisors see labeling that matches the ID field they are using.

Step 1 -  Open the Admin Panel inside ITpipes Web by clicking on the 3 horizontal lines in the upper right of the ITpipes Web interface, then select Administration

Step 2 -   Select "Project Administration", which should be the middle of the 3 tabs on the screen.

Step 3 -   Inside the Admin Panel (shown below) complete the necessary fields:


  1. The "Service" field is simply an alias to name the Feature Service.
  2. The URL is what points the secured service we talked about before. If ITpipes Web is cloud-hosted then a secure service is recommended. For internally hosted ITpipes Web that is not externally facing, an unsecured service would work fine.
  3. The AppID and Secret fields are also used for Username and Password for token-based authentication.
Step 4 -  After clicking connect, the Layers will appear below. The Asset Type drop-down next to each layer defines the asset type for that layer (manhole, mainline, lateral) and the ID field is the GIS field that uniquely identifies each asset of that type.


Step 5 -  Field Mapping: 
  1. Selecting the Paper and Pencil icon shows field mapping options from GIS to ITpipes. The picture below has the Sewerlines layer field mapping options.
  2. The fields on the left are pulled directly from that layer of the Feature Service. The dropdown box next to each GIS attribute will show a list of available ITpipes fields.
  3. This field mapping is used to populate ITpipes data directly form the GIS data.

  1. These are data-type specific! For example, if the unique identifier for the sewerlines layer (Facility_I shown below) is created as a number field, the only fields in the dropdown will also be number fields. Our field for unique identifiers (GIS ID shown below) is a character field. 
Step 6 -  Check Data Types:

Here is an example of data-type specificity:
Mapped Length and total segment length is a number field. Other ITpipes fields like Pipe Diameter show in the drop down

  1. For the Facility_I, ITpipes fields that include characters are shown:


Step 7 -  Map Display
  1. Once the feature service connection has been added, the map and layers appear side-by-side in ITpipes Web.
  2. This is what the map legend looks like with a small sample of the map. The "Default" section with a different color relates to ITpipes data that also exists in the GIS data.


  1. The map also allows a user in ITpipes Web to display GIS attribute data (but not edit)


Section 8 - How to create a hyperlink inside ArcMap

This process is for users who have ESRI's ArcMap desktop software, and want to create links in ArcMap layers that point to corresponding ITpipes inspections in the browser-based ITpipes Web software.  More information on using hyperlinks in ArcMap can be found here, ESRI's support documentation on Using Hyperlinks.  

To configure the hyperlinks in ArcMap for ITpipes inspections, follow the procedure below:

1. In the Map View, with the Table on Contents open, right-click on the layer that the hyperlink will be
created for. Go to Properties.




2. Select the Display tab in the properties (red arrow).



3. In the Hyperlinks section, check the box for "Enable Support Hyperlinks using field:” 
4. Select “Script” then click "Edit".
5. Under Parser. Select “Python”.



6. Replace the field in the OpenLink() to be the unique identifier field.
7. In the script window it will show an item called “path =”, delete the information that is currently there and input the following: “http://(website)/Asset/Search?assetType=ML&assetName=” (with quotations) then “+” (minus quotations)


9. Next, under the Fields box double click on the unique identifier field.
10. Click OK, then click OK again to close out the Layer Properties.
11. Use the hyperlink tool on the toolbar to select a line and a web page will open up the ITpipes Web Viewer with the associated inspection.  Note, there may be prompt to enter ITpipes Web credentials the first time.

Section 9 - How to create a hyperlink inside ArcGIS Pro

This process is for users who have ESRI's ArcGIS Pro desktop software, and want to create hyperlinks that point to corresponding ITpipes inspections in the browser-based ITpipes Web software.  

To add hyperlinks in ArcGIS Pro for ITpipes inspections, follow the procedure below.  For this procedure to work, it is required that the dataset have one field that is a designated asset ID field that matches the asset ID field in ITpipes.  ObjectID, FID, and GlobalID are not acceptable asset identification attributes.  

This link is a short video that goes step-by-step the instructions below:

1.  In ArcGIS Pro, or Catalog, open the attribute table for your manhole, mainline, or lateral data, add a new string field to your dataset.  The name of this new string attribute doesn't need to be specific, but in the example below the field was named "Hyperlink"



2.  Save the changes to your attribute table.  


3.  Populate this new string field with exactly what's shown below using the field calculator. 
            a.  To open the Field Calculator tool, right click on the column in the attribute table, and select Calculate Field
                    
            b.  In Field Calculator, type the text at the end of this step, or, as seen in the example here.  Click OK run the Field Calculator.  
                  
            c.  The red text in the example below should be replaced with the appropriate website domain.  In the example below,  demo.itpipes.com is the organization's website.
           d.  The green text in the example below should be replaced with the appropriate asset ID field in the attribute table.  In the example below, Facility_I is the attribute name of the unique asset ID field in the GIS attribute table.  

 "https://demo.itpipes.com/Asset/Search?assetType=ML&assetName="+!Facility_I!

3.  Now when an asset is selected in the map view, the new attribute field will automatically show up at the bottom of the popup.  When the link is clicked, ITpipes Web will open to the corresponding asset.

  

Section 10 - Creating Hyperlink in Web Map Viewer (ArcGIS Online)

  1. Verify that you are signed in. 
  2. Note: If privileges to create content are not authorized in the associated ESRI account,  pop-ups to explore data can still be configured by clicking Modify Map.
1. Open the Map Viewer for the Published map in ArcGIS Online.
2. Hover over the Mainline layer and select the three dots.   
3. Select Configure Pop-up to configure the pop-up window. Note: if the pop-ups are not enabled, it is required first to select Enable Pop-up.
4. Select Add under Attribute Expressions.
5. Select Edit at the top to rename-name the expression “ITpipes Web”.
6. Select Save to save the expression name.
7. Add the following (including quotes) on the blank line that exists:
  1. Return HTTP://{server}/Asset/Search?assetType=ML&assetName=” +
  2. Note: {server} needs to be replaced with the ITpipes Web URL. For example: if the ITpipes Web URL is ITpipesWeb.com. What will be placed is return “http://ITpipesWeb.com/Asset/Search?assetType=ML&assetName=” +
8. Under Global select the unique ID on the map. Note: The unique ID needs to match ITpipes' unique identifier.
9. Select OK twice to save the expression.
10. Last, it is required that the web map be SAVED, or these changes will be lost the next time this web map is opened.
11. When an asset is selected from the web map and the pop-up opens, the pop-up should not contain a clickable link.  This clickable link should open to the inspection in ITpipes Web which has a corresponding ID number.

Section 11 - GIS Map Testing

Once the map is setup inside of ITpipes Mobile it is recommended that it be tested to verify functionality

Do the following to do a map test inside ITpipes Mobile:
  1. Open the project the map was attached to.
  2. Confirm the map background is visible.
    1. If not, the tile package did not get attached or download correctly. In some rare instances this might be a result of PC graphics hardware incompatibility.
  3. Confirm the asset lines are visible and selectable.
    1. Lines should match the symbology originally published in the feature service.
  4. Send asset from the Map to ITpipes, and confirm field mapping is correct.
    1. Perform a test inspection from the created asset.
  5. Use the group selection and send multiple assets from the Map to ITpipes.
  6. Select the different assets inside of ITpipes to confirm the map is selecting and zooming to the assets.

Section 12 - Helpful ESRI Links

These links to ESRI articles are often valuable when working with ITpipes and ESRI GIS.


Section 13 - Troubleshooting & Tips

  • Lines appear large on the map viewer - Verify that any flow arrows are removed prior to publishing the feature service.

  • Flow Arrows not showing on map viewer - Follow the instructions that can be found here to add flow arrows.

  • Field mapping is not functioning correctly to pre-populate work orders - Please confirm a new project is used to test the field mapping, not an existing project.

  • Feature service will not download - 

    • This is typically due to connectivity issues.

    • Incorrect credentials.

    • Sync is not enabled on the feature service.

  • Plot does not function correctly

    • Verify that upstream and downstream are mapped in the field mapping for the mainline.  

    • Verify the coordinate system for the map is a WGS 1984 Web Mercator.

  • ITpipes mapping select tool is selecting the wrong asset and/or asset type on the map.

    • This is typically caused by assets not having a unique asset ID. Please confirm each asset, across all layers, has a unique asset ID.

  • ITpipes mapping select tool is not selecting an asset.

    • The unique ID in the map does not match the unique ID in the ITpipes database. This can be caused by assets having extra spaces at the end of the asset ID in either system or the id not matching any unique id on the map.


If you have any questions or need support beyond what this article offers, please reach out to us at (505) 341-0109, or email us at techsupport@itpipes.com.



    • Related Articles

    • ESRI Utility Network and/or Compliance Standard and ITpipes Templates

      Template Definition When a client starts performing inspections and using ITpipes, a template is defined that contains the fields for data input and the inputs to the fields. That template will have asset header fields that exist once for each asset, ...

    • GIS Map Testing for ITpipes Mobile

      Once the map is setup inside of ITpipes Mobile it is recommended that it be tested to verify functionality Do the following to do a map test inside ITpipes Mobile: Open the project the map was attached to. Confirm the map background is visible. If ...

    • GeoCortex + ITpipes Hyperlinking

      This KBA has details specific for GecCortex hyperlinking, limited integration function. Integration functions available: Hyperlink inside AMS to open ITpipes Web Viewer. With the Geocortex integration, a hyperlink is provided that will allow the user ...

    • ITpipes Education - User's Manual, Quickguides, Training Videos

      Users Manual Here is a link to ITpipes Users Manuals: ITpipes Web ITpipes Mobile ITpipes Sync Quickguides ITpipes Mobile ITpipes Sync Training Videos Links to available training videos.  Best when viewed full screen.   ITpipes Web  Creating a WO - ...

    • ITpipes Desktop Education - Users Manuals, Training Videos, Quickguides

      ITpipes Desktop/Legacy Product Important Announcement 1) ITpipes Desktop is no longer in active development (since pre-2018). ITpipes Desktop was developed using VB 6. ITpipes Desktop is recommended to be used on Microsoft Windows 7. Microsoft does ...