Blog

Basics of Data Science #DataScience #MachineLearning – Machine Learning Day 1

Data Science is all about understanding data, finding patterns and at the same time predict something out of it.

First step in data science is about getting hold of data, data can be available in different formats such as csv, excel, SQL tables, txt files etc.

Once we get data we can start exploring it and refining it as well. Excel is a very good tool to explore data which comes with lot of inbuilt functionalities.

Step two figure out any anomalies of data you have got and remove them such as

  • Duplicate data
  • Incorrect data such as most of the data ranges in 2 digits but suddenly you have a 3 digit or 4 digit data so you need to understand whether this is correct or should be removed
  • Blank data, any blank data can be filled with relevant data like  by taking an average or removing it.

Above list is not exhaustive but relevant, there can be other scenarios as well.

Once you have cleaned your data, third step is to use some methods to understand your data better

Using these methods you might be able to identify some trends or understand it better than before.

Once you have understood it better and you are smart enough you can ask following types of questions which forms basics of data science

  • Descriptive: Questions like how many?, how much ?
  • Associative: Relational data questions, like if temperature is high you sell more ice creams or you sell more ice creams between 2 PM to 4 PM
  • Comparative: Compare of flavor of ice creams being sold and compare them in a day or a month
  • Predictive: You can predict data such as chocolate flavor ice cream is most liked by children

If you can ask above questions it means you understand data quite well. Next step is to plot data in different ways such as Graphs, Charts, Histograms and also visualize data in 2D and 3D (Pivot table and charts)

Some common techniques are listed below

So these are some basics of data science, more in next blog

Data Science is huge subject which cannot be dealt in one blog, I have started a journey of acquiring a new skill “Machine Learning”, I am writing blog as I learn to keep notes for myself and for future learners. First day we learnt about basics of data science.

PowerShell to update an item in SharePoint List #SharePoint #PowerShell

If you are working in support you might get tickets to update a particular item as users are not able to update some columns as it’s locked due to some workflow or checks in forms´customization. The user in workflow is on leave or currently not available so as support you can do it from back via PowerShell.

It can be done via some simple PowerShell commands as below

$web = Get-SPWeb “Put site URL here”

$list = $web.Lists[“ListName”]

$item = $list.GetItemByID(46)

$item[“Status”] = “Approved”

$item.update()

So in above example first we create reference to the web we are going to work upon, then refer to list on which you are going to work upon.

Next step is to get a specific item by using ID and we are using GetItemByID function. As an example we have used item id 46, you need to replace this with your ID.

Next is to update a particular column value, here “Status” is a column name, which is being updated with new value “Approved”.

Last step “update” function is important else value will not be updated.

 

What is Azure Machine Learning ? #Azure #MachineLearning

Azure Machine Learning is a cloud based predictive analytics service which is easy to use and give fast and reliable results using its predefined algorithms and models.

Using Machine learning in broad / high level manner needs 4 steps described as below

  1. Data collection or generation
    • Collect data from source
    • Store in Azure Storage, SQL etc
    • Refine data if needed
  2. Create Machine Learning models
    • Use Machine Learning Azure Studio
    • Easy to use drag drop interface
  3. Output data as Web Service
    • Run the model to output data as web service
  4. Use output for intelligence or insights
    • Based on output make sites or Apps intelligent
    • Create insights and make predictions

Diagram for Azure Machine Learning

azuremachinelearning
Azure Machine Learning

Checklist for configuring eDiscovery in SharePoint Online #SharePoint #Office365 #eDiscovery

There are multiple steps which needs to be done to configure eDiscovery successfully in SharePoint Online or Office 365. To perform this you need to be Office 365 Tenant Admin.

Here is a checklist to confirm the set up for eDiscovery

  1. Have you created a Site Collection with template as eDiscovery Center?
  2. Have you configured Exchange Online as one of the Result Sources in eDiscovery Center?
  3. Have you created a Security group called eDiscovery Managers in Exchange Admin?
  4. Have you given access to security group eDiscovery Managers to Site Collection as Admin?
  5. Have you given Admin access to all Site Collections to security group eDiscovery Managers?
  6. Have you given read permissions to eDiscovery Managers to Crawl Log permissions in Search settings in SharePoint admin?
  7. Have you given access to Role Discovery Management in Exchange Admin to eDiscovery Managers?

If your answer to all above questions yes then you are ready and your eDiscovery Managers can create cases now.

 

Sideloading of Apps in #SharePoint online #Office365

If you are getting this error ” Error occurred in deployment step ‘Install SharePoint Add-in’: Sideloading of apps is not enabled on this site ” when debugging a SharePoint Add-in it means site you are using is a Production site. Sideloading of Apps is not enabled on any site by default as specially Production sites since it allow developers to load Apps which are still under development and not published.

If you are going to do some tests then enable it using PowerShell for SharePoint Online.

PowerShell code can be downloaded from MSDN code repository from Here

To disable Sideloading you can download code from here

 

 

Backup and restore strategy for SharePoint Online#SharePoint #Office365

SharePoint on-premise Backup and Restore is a big topic of its own but in SharePoint online you don’t have such options. On-prem version you could use PowerShell to do perform nightly jobs to take Backup or you can take SQL backups which can be used to restore later.

Nothing like this is possible in SharePoint Online, because you don’t have servers and there are no PowerShell commands to take back up or export a site , list or library.

So question is how we safeguard ourselves with scenarios which can happen and might have happened in on-prem.

  • User claims a file has been deleted as he / she can’t find it
  • User deletes a file and realizes after long time that he / she needs it back
  • Lot of permission settings have been changed and mistakes have been done, so how to see what it was like 1 day before so that same can be replicated again

Well what are the options with us

  1. Enable version history: Although it does not solve your backup and restore concerns but it can be helpful is certain scenarios such as user updated the document wrongly if version history is not enabled you end up losing earlier version. Of course you can ask Microsoft for a restore which they keep for 14 days but note that it is not a granular restore which means whole site collection is restored and you might have duplicates
  2. Use Recycle Bin: To lot of extent this solves a lot of worries of Backup and Restore. Each file, item , list or library, site deleted will be stored in Recycle Bin for 90 days. You can always restore your own files for 90 days and if deleted from own recycle bin it will be stored in site collection recycle bin. Also we have second stage recycle bin which can be accessed by site administrators. If a site collection is deleted then it can be restored within 30 days as it depends your Office 365 storage.
  3. Request Microsoft for a restore: Microsoft keeps 14 days backup and it can be restored by making a request to Microsoft. So if you do the math if a file was deleted 105 days back it can still be restored as you can go 14 days back with backup and considering that deleted file might be in Recycle bin so you can restore it from there. But this is not very ideal as you will have duplication of data and it can be only done on site collection level and both old and new files will be presented to user.

So what are other alternatives, you can go for some third-party tools which are available in market, they allow you to take backups, do granular restores much like you were doing in on-premise SharePoint but of course it drives cost due to licenses and storage costs in Azure or some other services or cheap in house storages.

Another thing I can think of is taking backup on your own and storing it in some cheap storage locally / in-house. I have not built any solution like this but it can be done as below

  • You already have Recycle bin and Microsoft 14 days backup which can save you in most of the scenarios
  • But what if there are some business critical data which you want to safeguard beyond what you are getting and you don’t want to spend more money on third party tools etc.
  • You can write a script using REST API which download data for business critical sites and stores locally. Download list data to excel. REST API supports downloading of data which can easily structured as backup if the script is written that ways. Frequency can be once a week so we have some extra cushion or daily depending on how long it takes to run, you can run parallel or multi threaded scripts to execute parallel.

If you like the idea of REST API let me know I can help you provide a solution based on that. Leave a comment and we can take it from there.

In case you need more help on this or other SharePoint issues please contact us via our Contact Us page or via our Facebook page or via Twitter

Like us on Facebook and follow us on Twitter

 

Hybrid Architecture Office 365 (SharePoint Online) and SharePoint 2016 On premise #SharePoint2016 #SharePoint #Office365

Best thing which has happened during the introduction of SharePoint 2016 is that Microsoft is convinced that Hybrid Architecture is the way forward for bigger organizations and all the data cannot be moved to cloud. At least this is the current reality with lot of companies but such fast changing technologies you never know what’s the future

In this post we will review architecture of SharePoint 2016 on-premise and Office 365 (SharePoint Online). Here we are assuming that you already have a SharePoint On-premise set up and you will add SharePoint Online.

Basic architecture looks like below

hybridarchitecture

Image source Download

To start with you need an Office 365 license and Azure subscription as well. To start with we need to set up server to server trust and certificates like STS.

We need Active Directory Federation Services (AD FS) which is used to share information securely and seamlessly. SSO (Single Sign on) is also facilitated via this and also makes sure that there no duplication of accounts.

Azure active directory tenant is used to authenticate to Office 365. It’s quiet powerful with variety of features like identity management, multifactor authentication etc. Since authentication occurs via Azure active directory we will have copy of all user account of organization in this as well but we will use Azure Active Directory connect  for directory synchronization so that for end-user it looks like only one account.

More about Azure Active Directory Connect can be read here Azure Active Directory Connect

Azure Access control services is used to provide users authentication and authorization to web application and services. You can read more here Azure Access Control Service

There are multiple steps to do the complete set up, if you need very detailed steps I will recommend below online books from Microsoft press store

Planning and Preparing for SharePoint Hybrid

Configuring SharePoint Hybrid capabilities

Once the set up is done your users will experience various benefits, but you need to be sure what you are getting into and be clear about it

  • Integrated search: You get integrated search so you can search in both SharePoint Online and SharePoint 2016 on-premise. Everything will be indexed in SharePoint online. Note that since index is stored in SharePoint some customers need to be wary of it they might not want part of secret data also in cloud. How to do set up can read here Configure Cloud Hybrid Search
  • Unified user profile where users can see both on-premise and SharePoint online followed sites together
  • Other services like Delve, Video and App launched will now be available on SharePoint on-premise as well
  • You can redirect users to SharePoint Online OneDrive for Business from SharePoint 2016 on-premise OneDrive. Before you do this make sure that all documents are migrated as it’s only a redirect. Steps to redirect are very simple and can be used read here How to redirect users to Office 365 for OneDrive for Business . Best way to migrate data is to write a tool for migration, there are various way to implement this but best is that you can access OneDrive for Business using SharePoint APIs which you might be familiar with else you have to register an App for OneDrive for Business to access data / folders programatically. I am assuming you already know how to access OneDrive in SharePoint on-premise programatically.

To access OneDrive for business SharePoint online via REST API you can below url

https://O365DomainHere-my.sharepoint.com/personal/     UserName_O365DomainHere_onmicrosoft_com/_api/web/GetFolderByServerRelativeUrl(‘/personal/UserName_O365DomainHere_onmicrosoft_com/Documents’)/folders/

You can read about data operations here REST API Data Operations SharePoint

So this is a basic overview of Hybrid architecture and also my take on the things you need to consider, if you have further questions just leave  a comment

In case you need more help on this or other SharePoint issues please contact us via our Contact Us page or via our Facebook page or via Twitter

Like us on Facebook and follow us on Twitter

 

SharePoint 2013 migration to Office 365 step by step – Data Migration #SharePoint #Office365

After building the strategy and planning for it, next step is Data Migration from SharePoint 2013 to Office 365. We are going to look into native import / data migration service provided in Office 365, you need to be an Office 365 Administrator or at least have someone to help with initial set up.

Before we start actual migration we need to do some set up. Login to Office 365 and go to menu Users –> Migration.

datamigration

Once there scroll to see Onboarding and migration guidance, click on SharePoint and new window will open.

onboardingmeny

This will guide to Onboard you and create migration jobs, I am just capturing relevant steps here

onboardingsteos

Step 1 is just some text about SharePoint online

Step 2 is very important as you need to define what kind of deployment you will have, in our case we will select on premise to cloud.

oboarding-step-2

Step 3 explains PowerShell option we have selected On-premise to cloud migration

Step 4 is to prepare for PowerShell, first it download and install SharePoint Online Management Shell, download it from here SharePoint Online Management Shell.

Second is to create Azure Storage account if you don’t have already one.

Go to https://portal.azure.com and create a storage account

storageaccount

Also get your location and details

step-4

sponlinepowershell

Step 5 is creation of Content package, we need to use classic Export Web command in SharePoint onpremise server to create an export package

Export-SPWeb [-Identity] -Path [-ItemUrl ] [-NoFileCompression ]

Step 6 is coversion of package so that it can be imported

ConvertTo-SPOMigrationTargetedPackage -SourceFilesPath $sourceFiles -SourcePackagePath $sourcePackage -OutputPackagePath $targetPackage -TargetWebUrl $targetWeb -TargetDocumentLibraryPath $targetDocLib -Credentials $creds

To read more about ConvetTo-SPOMigrationTargetedPackage you can read here Click here

Step 7 is creating Azure Containers and uploading content package

$al = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFiles -SourcePackagePath $targetPackage -AzureQueueName $azureQueueName -AccountName $azureAccountName -AccountKey $azureAccountKey

$al|$al|fl

Step 8 is submitting a migration job, sample code is here

Submit-SPOMigrationJob -TargetWebUrl $targetWeb -MigrationPackageAzureLocations $al -Credentials $creds

Note that some parameters variables were created in earlier steps and needs to be used here

Step 9 is to check Migration Job Status which can be done Azure account storage queue

Sample source code

# Example file share packaging, conversion, upload and job submission script
# Queue is only container specified as new containers are automatically generated
# All container URIs get populated into $al variable output from Set-SPOMigrationPackageAzureSource cmdlet
$creds = (Get-Credential admin@contoso.com)
$sourceFiles = ‘\\fileshare\users\charles’
$sourcePackage = ‘C:\migration\CharlesDocumentsPackage_source’
$targetPackage = ‘C:\migration\CharlesDocumentsPackage_target’
$targetWeb = ‘https://contoso-my.sharepoint.com/personal/charles_contoso_com’
$targetDocLib = ‘Documents’
$azureAccountName = ‘contosomigration’
$azureAccountKey = ‘Insert Azure Account Key string Here’
$azureQueueName = ‘migrationqueue’
# Create new package from file share without security (faster)
New-SPOMigrationPackage -SourceFilesPath $sourceFiles -OutputPackagePath $sourcePackage
# Convert package to a targeted one by looking up data in target site collection
ConvertTo-SPOMigrationTargetedPackage -SourceFilesPath $sourceFiles -SourcePackagePath $sourcePackage -OutputPackagePath $targetPackage -TargetWebUrl $targetWeb -TargetDocumentLibraryPath $targetDocLib -Credentials $creds
# Create azure containers and upload package into them, finally snapshotting all files
$al = Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFiles -SourcePackagePath $targetPackage -AzureQueueName $azureQueueName -AccountName $azureAccountName -AccountKey $azureAccountKey
# This displays the return Azure location
$al|$al|Format-List
# Submit package data to site collection to create new migration job
Submit-SPOMigrationJob -TargetWebUrl $targetWeb -MigrationPackageAzureLocations $al -Credentials $creds

So this completes all steps for data migration. Next we will look into Hybrid scenarios which will come in next post

To check earlier two posts click below

SharePoint 2013 migration to Office 365 step by step – Planning #SharePoint #Office365

SharePoint 2013 migration to Office 365 step by step – Strategy #SharePoint #Office365

 

 

 

SharePoint 2013 migration to Office 365 step by step – Strategy #SharePoint #Office365

This post is written as step 2 that is strategy. Step that is planning can be read here SharePoint 2013 migration to Office 365 step by step – Planning #SharePoint #Office365

Once we have filled in the excel template and also found out all issues via SharePoint Migration assessment Tool we need to formulate a strategy to find solution to issues and customizations.

As an outcome of the first step you should have a list of categorization of issues / customizations for which you need to strategize. I am listing down some if it, note that these are just examples and it totally depends on what you have done in your farm before and how much customizations have been done.

customizationcategory

For more reading about above topics and in text format use below link to download an excel Customization / Issues Categorization

Note that any solution will be based on what kind of Office 365 license you have, whether your organization will have PowerApps or Flow or not? Also whether you will have SharePoint Add-in model in place or not

If you have any questions with specific customization solution you can ask in comment section and I will try my best to provide a solution to that.

After we define a strategy for customizations we need to find a strategy to migrate data from On-premise to Office 365. This will be our third step and we will define that in next post. Before we do data migration it’s important to list down how many Web Applications we have, size of Content Databases.

In case you need more help on this or other SharePoint issues please contact us via our Contact Us page or via our Facebook page or via Twitter

Like us on Facebook and follow us on Twitter

 

SharePoint 2013 migration to Office 365 step by step – Planning #SharePoint #Office365

This post is written as step 1 that is planning.

When SharePoint 2010 was launched Microsoft wanted to move everyone to cloud and they were trying to convince everybody that cloud is the future but they soon realized everything cannot be moved to cloud due to local data regulations and customers apprehensions about data security. Now Hybrid architecture is promoted quite a lot which makes sense and also more practical.

In this post I will try to outline how to prepare for Migration from SharePoint 2010 / 2013 to Office 365. We also discuss Hybrid architecture later in other posts.

We will discuss different aspects of migration, in this post we will understand SMAT tool provided by Microsoft and also excel template created by me to analyze customizations.

Microsoft has introduced a SharePoint Migration assessment Tool called SMAT, you can download it from here SMAT.

The zip file downloaded needs to be run on a SharePoint server, file contains below files, some of them are important to understand

smatfiles

File SiteSkipList.csv is important as you can mention which sites needs to be skipped during assessment, like you can mention MySite to be skipped as in general they don’t have any customizations and many sites under it.

SMAT Read me have more instructions but to run it you need run the exe file via command prompt

SMAT.exe -0 D:\Outputlog

where D:\Outputlog is the destination where logs will be created, do not out the log destination as SMAT folder as it tries to clean up and will end up deleting files.

While its running you can see what it will be scanning, below is a screenshot while it runs. You can see it’s analyzing FullTrustSolutions, LargeExcelFiles, Wokflows etc.

Since we know that no server side code is allowed in Office365 it’s important to have this kind of analysis to understand what things needs to reimplemented.

smatcapture

Once the assessment tool finished it will generate reports for each topic in detail and list down all site details, issues about it like below

smatreports

Each report has very detailed information, if you want to understand meaning of each report you can refer to this link SMAT Report Definitions. Although I think names are self-explanatory.

These reports will help you understand what issues you will face if you need to migrate but for planning I have created excel template where you can list all customizations and also strategize around it. This is needed because I believe that no one understands your farm better.

You can download the excel template from here, each column has some explaination.There are two sheets. First sheet is for listing all customizations and categorized them. Second sheet is to write down solution details of each category. This way you can have common solutions for each category and it can define how your developers should approach re-implementation

SharePoint 2013 to Office 365 Migration Template

So the migration assessment tool and excel template will help you analyze your farm in great detail and plan for the migration.

Tech Tip : Are you in search of a remote working technology? Access your online private workspace on any device(PC/android/iOS) with remotely accessible citrix vdi from CloudDesktopOnline with enterprise Add-Ons like Hosted SharePoint and Exchange. If you`re a MS Azure user looking for managed azure services visit Apps4Rent.

You can read the part 2 here SharePoint 2013 migration to Office 365 step by step – Strategy #SharePoint #Office365