Update ISIN Data

Bumps [mako](https://github.com/sqlalchemy/mako) from 1.2.0 to 1.2.2.
- [Release notes](https://github.com/sqlalchemy/mako/releases)
- [Changelog](https://github.com/sqlalchemy/mako/blob/main/CHANGES)
- [Commits](https://github.com/sqlalchemy/mako/commits)

---
updated-dependencies:
- dependency-name: mako
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

.github/FUNDING.yml

@@ -0,0 +1,3 @@
+github: captn3m0
+ko_fi: captn3m0
+liberapay: captn3m0

.github/workflows/update.yml

@@ -19,6 +19,7 @@ jobs:
       id: update_data
     # Only tag if we're running on the scheduled job
     - uses: stefanzweifel/git-auto-commit-action@v4
+      id: auto-commit-action
       with:
         commit_message: Update ISIN Data
         commit_author: 'github-actions[bot] <github-actions[bot]@users.noreply.github.com>'
@@ -26,6 +27,7 @@ jobs:
         status_options: '--untracked-files=no'
         tagging_message: "v${{ steps.update_data.outputs.version }}"
     - name: Create Release
+      if: steps.auto-commit-action.outputs.changes_detected == 'true'
       run: |
         pip install -r src/requirements.txt
         make release "version=v${{ steps.update_data.outputs.version }}"

.gitignore

@@ -1,4 +1,5 @@
 pup
 release.md
 pup.zip
-IN*.csv
+IN*.csv
+notes.md

CITATION.cff

@@ -5,9 +5,9 @@ authors:
   given-names: "Abhay"
   orcid: https://orcid.org/0000-0002-7993-1363
 title: "India ISIN Database, by Nemo"
-version: 2022.5.19
+version: 2024.6.11
 url: "https://github.com/captn3m0/india-isin-data"
-date-released: 2022-05-19
+date-released: 2024-06-11
 preferred-citation:
   url: "https://github.com/captn3m0/india-isin-data"
   type: data

Makefile

@@ -30,8 +30,8 @@ update: ISIN
 old:
 	git show HEAD^:ISIN.csv > /tmp/ISIN.csv
 
-release.md: old
-	python3 src/diff.py > release.md
+release-notes: old
+	python3 src/diff.py
 
-release: release.md
-	gh release create "$(version)" --notes-file release.md ISIN.csv
+release: release-notes
+	gh release create "$(version)" --notes-file notes.md ISIN.csv release.md

README.md

@@ -2,7 +2,7 @@
 
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6508187.svg)](https://doi.org/10.5281/zenodo.6508187) ![GitHub release (latest by date)](https://img.shields.io/github/v/release/captn3m0/india-isin-data) ![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/captn3m0/india-isin-data) ![GitHub repo size](https://img.shields.io/github/repo-size/captn3m0/india-isin-data) [![Flat GitHub Badge](https://img.shields.io/badge/View_Data_on-Flat_Github-GREEN.svg)](https://flatgithub.com/captn3m0/india-isin-data)
 
-ISIN Data from various public securities. You can sort and filter this dataset in your browset at https://flatgithub.com/captn3m0/india-isin-data.
+ISIN Data from various public securities. You can sort and filter this dataset in your browser at https://flatgithub.com/captn3m0/india-isin-data.
 
 Source: [NSDL Website Detailed ISIN Search][nsdl].
 
@@ -12,24 +12,25 @@ Automatically updated every midnight (IST).
 
 # ISIN Structure
 
-ISIN (Internation Security Identification Numbering System) is defined by [ISO 6166:2021][iso] and adopted in India by BIS as [IS 15415:2021][bis] ([(PDF)][bispdf]) ([(PDF2)](https://archive.org/details/gov.in.is.15415.2003)). ISINs in India are [alloted by NSDL for all securities except Government Securities](https://investor.sebi.gov.in/pdf/reference-material/ppt/PPT-7%20Depository%20Services.pdf) (See Page 9).
+ISIN (International Securities Identification Number) is defined by [ISO 6166:2021][iso] and adopted in India by BIS as [IS 15415:2021][bis] ([(PDF)][bispdf]) ([(PDF2)](https://archive.org/details/gov.in.is.15415.2003)). ISINs in India are [allotted by NSDL for all securities except Government Securities](https://investor.sebi.gov.in/pdf/reference-material/ppt/PPT-7%20Depository%20Services.pdf) (See Page 9).
 
 > This Indian Standard (Second Revision) which is identical with ISO 6166 : 2021 ‘Financial services - International securities identification number (ISIN)’ issued by the International Organization for Standardization (ISO) was adopted by the Bureau of Indian Standards on recommendation of the Banking and Financial Services Sectional Committee and approval of the Services Sector Division Council.
 
 ISIN, as per the ISO standard consists of 3 parts:
 
-1. 2 Character country code as prefix (ISO-3166). For ISINs issued in India, this will always be `IN`.
-2. The Basic Number, 9 alphanumeric characters left padded with zeroes.
-3. A check digit.
+1. **A 2 Character country code as prefix (ISO-3166)**. For ISINs issued in India, this will always be `IN`.
+2. **A Basic Identification Number**. 9 alphanumeric characters left padded with zeroes.
+3. **A check digit**.
 
 The basic number in India is issued by NSDL and is composed of 4 parts:
 
-1. 1 Character for the kind of issuer. This varies from A-F/9.
-1. A 4 character Issuer Code. Alphanumeric.
-1. A 2 Character code for security type. Alphanumeric.
-1. A 2 Digit serial number for the security itself. Alphanumeric.
+1. A 1-character **Issuer-Type**. This varies from A-F/9.
+1. A 4-character **Issuer code**. Alphanumeric.
+1. A 2-character **Security Type**. Alphanumeric.
+1. A 2-digit **serial number for the security itself**. Alphanumeric.
 
-Hence, each issuer can issue 36^2 = 1296 securities under each type. The table below explains the same with an example:
+Hence, each issuer can issue 36^2 = 1296 securities under each type.  
+The table below explains the same with an example:
 
 <div>
 <table>
@@ -62,7 +63,7 @@ Hence, each issuer can issue 36^2 = 1296 securities under each type. The table b
             <th>Class</th>
 			<td style="text-align: center"><code>I</code></td>
 			<td style="text-align: center"><code>N</code></td>
-			<td style="text-align: center"><code>A-F,9</code></td>
+			<td style="text-align: center"><code>A-F,0-9</code></td>
 			<td style="text-align: center"><code>alpha</code></td>
 			<td style="text-align: center"><code>alpha</code></td>
 			<td style="text-align: center"><code>alpha</code></td>
@@ -93,7 +94,7 @@ Hence, each issuer can issue 36^2 = 1296 securities under each type. The table b
 </table>
 </div>
 
-In the above ISIN (`INF789F01XA0`), here's the breakdown:
+Here's the breakdown of the above ISIN (`INF789F01XA0`):
 
 - Country Code: `IN`
 - Issuer Type: `F` (Mutual Funds)
@@ -102,17 +103,17 @@ In the above ISIN (`INF789F01XA0`), here's the breakdown:
 - Serial Number: `XA`
 - Check Digit: `0`
 
-### Isser Types
+### Issuer Types
 
-Issuer Type|Isser Type Code
+Issuer Type|Issuer Type Code
 -----------|-------------
 Central Government|`A`
 State Government|`B`
 Municipal Corporation|`C`
 Union Territories|`D`
-Company, Statuatory Corporation, Banking Company|`E`
+Company, Statutory Corporation, Banking Company|`E`
 Mutual Funds|`F`
-Partly Paidup Shares|`G`
+Partly Paid-up Shares|`G`
 Central Government Security|`0`
 State Government Security|`1`
 State Government Security|`2`
@@ -169,7 +170,7 @@ Mutual Fund (`INF`) | C1 |   Mutual Fund Unit
 
 ### Check Digit
 
-The last digit (check-digit) is calculated using the [Luhn Algorithm](https://en.wikipedia.org/wiki/Luhn_algorithm) with a slight change to accomodate alphanumeric characters. Alphabets are converted to digits by adding `9` to the usual numeric value of each letter. For example `A=10,B=12,C=13……Z=35`. A few examples:
+The last digit (check-digit) is calculated using the [Luhn Algorithm](https://en.wikipedia.org/wiki/Luhn_algorithm) with a slight change to accommodate alphanumeric characters. Alphabets are converted to digits by adding `9` to the usual numeric value of each letter. For example `A=10, B=12, C=13, ..., Z=35`. A few examples:
 
 ISIN|Payload|Check Digit|Validate
 ----|-------|--------|---
@@ -213,6 +214,8 @@ ISINs for India can be found at a few other sources:
 - The [OpenFIGI API](https://www.openfigi.com/api) returns results for some (not all) Indian ISINs.
 - The National Stock Exchange has [a few](https://www1.nseindia.com/products/content/debt/wdm/gsec_reporting_homepage.htm) [pages](https://www1.nseindia.com/products/content/debt/ncbp/ncbp_issues.htm) listing Government Security ISINs.
 - Similarly, the Reserve Bank of India also lists Government Securities at a few pages: [[1]](https://www.rbi.org.in/Scripts/bs_viewcontent.aspx?Id=3876), [[2]](https://rbi.org.in/Scripts/bs_viewcontent.aspx?Id=1956), [[3](https://rbi.org.in/scripts/Bs_viewcontent.aspx?Id=3973)], [[4](https://rbi.org.in/scripts/BS_PressReleaseDisplay.aspx?prid=51712#AN1)], [[5](https://rbi.org.in/scripts/BS_PressReleaseDisplay.aspx?prid=52077)]. 
+- [Securities admitted for Dematerialisation](https://nsdl.co.in/securities_admitted_for_dematerialisation.php) on the NSDL website
+- [CSV Download](https://nsdl.co.in/list-codes.php) on the NSDL website. This doesn't include [Government securities](https://forum.fossunited.org/t/indias-isin-dataset/1090/4?u=nemo)
 
 # License
 

datapackage.json

@@ -78,6 +78,6 @@
     }
   ],
   "title": "India ISIN Database",
-  "version": "2022.5.19",
-  "created": "2022-05-19 21:25:05+00:00"
+  "version": "2024.6.11",
+  "created": "2024-06-11 19:08:50+00:00"
 }

src/diff.py

@@ -12,9 +12,29 @@ def chunks(lst, n):
         yield lst[i:i + n]
 
 diff = compare(
-    load_csv(open("/tmp/ISIN.csv" % i), key="ISIN"),
-    load_csv(open("ISIN.csv" % i), key="ISIN"),
+    load_csv(open("/tmp/ISIN.csv"), key="ISIN"),
+    load_csv(open("ISIN.csv"), key="ISIN"),
     True
 )
 
-print(template.render(added=diff['added'], changed=diff['changed'], removed=diff['removed']))
+contents = template.render(added=diff['added'], changed=diff['changed'], removed=diff['removed'])
+
+# GitHub supports a maximum limit of 125000 for release notes.
+# 124800 = 124000 - 200 for buffer and the warning line
+# To get around this, we add a warning if we hit the limit
+# and attach a release.md file with the complete notes
+if len(contents) >= 125000:
+    notes = """This file is truncated due to GitHub limitations.
+please see the attached `release.md` file for complete notes""" + contents[:124800]
+else:
+    notes = contents
+
+"""
+notes are the viewable notes on the release page on GitHub
+while release.md is the attached notes. This is the complete text.
+"""
+with open('notes.md', 'w') as f:
+    f.write(notes)
+
+with open('release.md', 'w') as f:
+    f.write(contents)

src/fetch.sh

@@ -24,11 +24,11 @@ function fetch_page() {
     --data cnum=$1 \
     --data "page_no=$2" | \
   $PUP_BINARY '#nsdl-tables tr json{}' | \
-  # generate 6 lines (second column has a link, so parse that) with raw output
-  jq --raw-output '.[] | [.children[1].children[0].text, .children[2].text, .children[3].text,.children[4].text,.children[5].text]|.[]' | \
-  # and create a CSV from every 5 lines
-  paste -d, - - - - -  | \
-  # and we don't need the first row
+  # Generate a CSV (this contains the header row as well)
+  jq --raw-output '.[] | [.children[1].children[0].text, .children[2].text, .children[3].text,.children[4].text,.children[5].text]|@csv' | \
+  # Convert & to &
+  sed 's/&/\&/g' | \
+  # Drop the first row
   tail -n +2 >> "$3"
 }
 function fetch_total_pages() {
@@ -61,4 +61,5 @@ sem --wait
 sort -o "$CLASS.csv" "$CLASS.csv"
 # Remove lines that don't start with the correct prefix
 # This is to avoid ISINs like INF955L01IN9 showing up under IN9
-sed -i "/^$CLASS/!d" "$CLASS.csv"
+# Note that there is a " at the beginning to account for quoted CSVs
+sed -i "/^\"$CLASS/!d" "$CLASS.csv"

src/release.mako

@@ -1,6 +1,5 @@
 ## Generates a single diff for a single row, ignoring the Status field
-<%def name="singlediff(row)">
-@@ ${row['key']}
+<%def name="singlediff(row)">@@ ${row['key']}
 %for k in row['changes']:
 %if k != 'Status':
 -${k}:${row['changes'][k][0]}
@@ -19,6 +18,7 @@ ${row['changes']['Description'][1]}
 %endif
 </%def>
 
+% if len(added) > 0:
 <h2>Additions</h2>
 
 The following new ISINs were added:
@@ -29,6 +29,10 @@ ISIN|Description|Issuer|Type|Status
 `${row['ISIN']}`|${row['Description']}|${row['Issuer']}|${row['Type']}|${row['Status'].title()}
 % endfor
 
+%endif
+
+
+% if len(changed) > 0:
 <h2>Changes</h2>
 
 The following ISINs changed their status:
@@ -56,8 +60,6 @@ The following ISINs changed their status:
 </tbody>
 </table>
 
-## This will usually contain the description
-
 The following ISINs changed other fields:
 
 ```diff
@@ -68,7 +70,9 @@ ${singlediff(row)}
 % endfor
 ```
 
-## Removals are currently happening accidentally because NSDL website returns a 5xx
+%endif
+
+## Removals may happen accidentally because NSDL website returns a 5xx
 
 %if len(removed) > 0:
 
@@ -82,4 +86,4 @@ ISIN|Description|Issuer|Type|Status
 `${row['ISIN']}`|${row['Description']}|${row['Issuer']}|${row['Type']}|${row['Status'].title()}
 % endfor
 
-%endif
+%endif

src/requirements.txt

@@ -1,6 +1,6 @@
 click==8.1.3
 csv-diff==1.1
 dictdiffer==0.9.0
-Mako==1.2.0
+Mako==1.2.2
 MarkupSafe==2.1.1
 six==1.16.0